2024/11/22-23:13:02.069736 140024986822208 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00: fork to child time = 0.013098 sec 2024/11/22-23:13:02.070684 140024986822208 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00 2024/11/22-23:13:02.077820 140024986822208 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00 2024/11/22-23:13:02.079005 140024986822208 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-23:13:02.079272 140024986822208 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-23:13:02.135590 140024986822208 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-13679/att-00/MANIFEST-206347 2024/11/22-23:13:02.152892 140024986822208 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-13679/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 233196, last_sequence is 3634561247, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 233055 2024/11/22-23:13:02.152901 140024986822208 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-23:13:02.152986 140024986822208 [version_set.cc:5721] Creating manifest 233196 2024/11/22-23:13:02.185024 140024986822208 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-23:13:02.205590 140024986822208 [compaction_job.cc:2437] [default] [JOB 13679] Compacting 1@2 + 15@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/22-23:13:02.205610 140024986822208 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 10260 Base level 2, inputs: [233192(27MB)], [233141(34MB) 233142(35MB) 233143(35MB) 233144(35MB) 233145(35MB) 233146(34MB) 233147(34MB) 233148(33MB) 233149(34MB) 233150(37MB) 233151(35MB) 233152(36MB) 233153(34MB) 233154(37MB) 233156(35MB)] 2024/11/22-23:13:02.213260 140024986822208 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-23:13:02.213293 140024986822208 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-23:13:02.213313 140024986822208 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-23:13:02.213318 140024986822208 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-23:13:02.213329 140024986822208 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-23:13:02.691488 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233197.sst) /dev/shm/ToplingTemp/Topling-328270-x30aHs 2024/11/22-23:13:03.456097 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233197.sst): first pass time = 0.76's, 175.547'MB/sec 2024/11/22-23:13:05.755154 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233197.sst): old prealloc_size = 147639500, real_size = 32419744 2024/11/22-23:13:05.755835 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233197.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925072 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0304 GB avg-val = 117.09 avg-zval = 32.89 2024/11/22-23:13:05.756652 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233197.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:06.872090 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233197.sst): second pass time = 2.30's, 47.144'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.12's, 32.587'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2666.384'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.240'MB/sec write SST data time = 0.15's, 236.416'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34012.713'MB/sec zip my value time = 2.30's, unzip length = 0.108'GB zip my value throughput = 47.144'MB/sec zip pipeline throughput = 94.604'MB/sec entries = 925072 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.09 avg-zval = 32.89 usrkeys = 925072 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.09 avg-zval = 32.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925072 bytes = 3122128 } seq expand size = 6475504 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5133 value = 3.5597 dict = 1.00 all = 3.6876 } Zip/UnZip{ index = 0.0951 value = 0.2809 dict = 1.00 all = 0.2712 } ---------------------------- total value len = 0.114791 GB avg = 0.124 KB (by entry num) total key len = 0.025902 GB avg = 0.028 KB total ukey len = 0.018501 GB avg = 0.020 KB total ukey num = 0.000925072 Billion total entry num = 0.000925072 Billion write speed all = 0.002724304 MB/sec (with seq num) write speed all = 0.002581003 MB/sec (without seq num) 2024/11/22-23:13:06.907436 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233197: 925072 keys, 36397322 bytes, temperature: kWarm 2024/11/22-23:13:06.907518 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288386907469, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233197, "file_size": 36397322, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3513012344, "largest_seqno": 3634539567, "table_properties": {"data_size": 30428496, "index_size": 1759808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25902016, "raw_average_key_size": 28, "raw_value_size": 108315812, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925072, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288382, "oldest_key_time": 0, "file_creation_time": 1732288382, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233197, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:06.950256 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233198.sst) /dev/shm/ToplingTemp/Topling-328270-1C8TZr 2024/11/22-23:13:07.996944 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233198.sst): first pass time = 1.05's, 128.235'MB/sec 2024/11/22-23:13:09.673430 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233198.sst): old prealloc_size = 147639500, real_size = 33179840 2024/11/22-23:13:09.674124 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233198.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924861 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0312 GB avg-val = 117.12 avg-zval = 33.70 2024/11/22-23:13:09.674949 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233198.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:10.635641 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233198.sst): second pass time = 1.68's, 64.644'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 38.737'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2646.806'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.287'MB/sec write SST data time = 0.16's, 237.965'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 25344.477'MB/sec zip my value time = 1.68's, unzip length = 0.108'GB zip my value throughput = 64.644'MB/sec zip pipeline throughput = 80.665'MB/sec entries = 924861 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.12 avg-zval = 33.70 usrkeys = 924861 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.12 avg-zval = 33.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924861 bytes = 3237024 } seq expand size = 6474027 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.3766 value = 3.4757 dict = 1.00 all = 3.6010 } Zip/UnZip{ index = 0.0964 value = 0.2877 dict = 1.00 all = 0.2777 } ---------------------------- total value len = 0.229587 GB avg = 0.124 KB (by entry num) total key len = 0.051798 GB avg = 0.028 KB total ukey len = 0.036999 GB avg = 0.020 KB total ukey num = 0.001849933 Billion total entry num = 0.001849933 Billion write speed all = 0.005448181 MB/sec (with seq num) write speed all = 0.005161634 MB/sec (without seq num) 2024/11/22-23:13:10.668068 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233198: 924861 keys, 37272662 bytes, temperature: kWarm 2024/11/22-23:13:10.668128 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288390668097, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233198, "file_size": 37272662, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038037, "largest_seqno": 3634537942, "table_properties": {"data_size": 31165872, "index_size": 1782592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25896108, "raw_average_key_size": 28, "raw_value_size": 108321674, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924861, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288386, "oldest_key_time": 0, "file_creation_time": 1732288386, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233198, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:10.704528 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233199.sst) /dev/shm/ToplingTemp/Topling-328270-smUxzs 2024/11/22-23:13:11.740707 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233199.sst): first pass time = 1.04's, 129.536'MB/sec 2024/11/22-23:13:13.490392 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233199.sst): old prealloc_size = 147639500, real_size = 32722560 2024/11/22-23:13:13.491089 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233199.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925164 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0307 GB avg-val = 117.07 avg-zval = 33.22 2024/11/22-23:13:13.491888 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233199.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:14.524917 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233199.sst): second pass time = 1.75's, 61.932'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 35.588'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2608.745'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.294'MB/sec write SST data time = 0.25's, 149.468'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33774.555'MB/sec zip my value time = 1.75's, unzip length = 0.108'GB zip my value throughput = 61.932'MB/sec zip pipeline throughput = 83.453'MB/sec entries = 925164 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.07 avg-zval = 33.22 usrkeys = 925164 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.07 avg-zval = 33.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925164 bytes = 3238096 } seq expand size = 6476148 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.5132 value = 3.5246 dict = 1.00 all = 3.6456 } Zip/UnZip{ index = 0.0951 value = 0.2837 dict = 1.00 all = 0.2743 } ---------------------------- total value len = 0.344376 GB avg = 0.124 KB (by entry num) total key len = 0.077703 GB avg = 0.028 KB total ukey len = 0.055502 GB avg = 0.020 KB total ukey num = 0.002775097 Billion total entry num = 0.002775097 Billion write speed all = 0.008171682 MB/sec (with seq num) write speed all = 0.007741863 MB/sec (without seq num) 2024/11/22-23:13:14.561012 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233199: 925164 keys, 36815936 bytes, temperature: kWarm 2024/11/22-23:13:14.561103 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288394561048, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233199, "file_size": 36815936, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3487828160, "largest_seqno": 3634539280, "table_properties": {"data_size": 30731056, "index_size": 1760000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25904592, "raw_average_key_size": 28, "raw_value_size": 108313164, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925164, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288390, "oldest_key_time": 0, "file_creation_time": 1732288390, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233199, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:14.637543 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233200.sst) /dev/shm/ToplingTemp/Topling-328270-vaysZp 2024/11/22-23:13:15.541227 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233200.sst): first pass time = 0.90's, 148.531'MB/sec 2024/11/22-23:13:17.361687 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233200.sst): old prealloc_size = 147639500, real_size = 33494576 2024/11/22-23:13:17.362237 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233200.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924943 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.11 avg-zval = 34.06 2024/11/22-23:13:17.363018 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233200.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:18.328847 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233200.sst): second pass time = 1.82's, 59.524'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 38.864'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3304.209'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.276'MB/sec write SST data time = 0.13's, 290.144'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 28282.799'MB/sec zip my value time = 1.82's, unzip length = 0.108'GB zip my value throughput = 59.524'MB/sec zip pipeline throughput = 118.757'MB/sec entries = 924943 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.11 avg-zval = 34.06 usrkeys = 924943 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.11 avg-zval = 34.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924943 bytes = 3237312 } seq expand size = 6474601 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5168 value = 3.4383 dict = 1.00 all = 3.5708 } Zip/UnZip{ index = 0.0951 value = 0.2908 dict = 1.00 all = 0.2800 } ---------------------------- total value len = 0.459170 GB avg = 0.124 KB (by entry num) total key len = 0.103601 GB avg = 0.028 KB total ukey len = 0.074001 GB avg = 0.020 KB total ukey num = 0.003700040 Billion total entry num = 0.003700040 Billion write speed all = 0.010894761 MB/sec (with seq num) write speed all = 0.010321725 MB/sec (without seq num) 2024/11/22-23:13:18.367451 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233200: 924943 keys, 37587622 bytes, temperature: kWarm 2024/11/22-23:13:18.367508 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288398367478, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233200, "file_size": 37587622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038075, "largest_seqno": 3634539656, "table_properties": {"data_size": 31504160, "index_size": 1758976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25898404, "raw_average_key_size": 28, "raw_value_size": 108319382, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924943, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288394, "oldest_key_time": 0, "file_creation_time": 1732288394, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233200, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:18.416963 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233201.sst) /dev/shm/ToplingTemp/Topling-328270-J4K8Nq 2024/11/22-23:13:19.224438 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233201.sst): first pass time = 0.81's, 166.226'MB/sec 2024/11/22-23:13:20.821144 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233201.sst): old prealloc_size = 147639500, real_size = 33421664 2024/11/22-23:13:20.821674 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233201.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925225 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 117.06 avg-zval = 33.97 2024/11/22-23:13:20.822461 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233201.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:21.768625 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233201.sst): second pass time = 1.60's, 67.882'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.473'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3481.417'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 285.480'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33961.482'MB/sec zip my value time = 1.60's, unzip length = 0.108'GB zip my value throughput = 67.882'MB/sec zip pipeline throughput = 132.151'MB/sec entries = 925225 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.06 avg-zval = 33.97 usrkeys = 925225 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.06 avg-zval = 33.97 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925225 bytes = 3122656 } seq expand size = 6476575 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5112 value = 3.4461 dict = 1.00 all = 3.5887 } Zip/UnZip{ index = 0.0951 value = 0.2902 dict = 1.00 all = 0.2786 } ---------------------------- total value len = 0.573958 GB avg = 0.124 KB (by entry num) total key len = 0.129507 GB avg = 0.028 KB total ukey len = 0.092505 GB avg = 0.020 KB total ukey num = 0.004625265 Billion total entry num = 0.004625265 Billion write speed all = 0.013617573 MB/sec (with seq num) write speed all = 0.012901292 MB/sec (without seq num) 2024/11/22-23:13:21.815176 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233201: 925225 keys, 37399760 bytes, temperature: kWarm 2024/11/22-23:13:21.815234 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288401815204, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233201, "file_size": 37399760, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3523718385, "largest_seqno": 3634537835, "table_properties": {"data_size": 31429712, "index_size": 1760448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25906300, "raw_average_key_size": 28, "raw_value_size": 108311450, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925225, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288398, "oldest_key_time": 0, "file_creation_time": 1732288398, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233201, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:21.862006 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233202.sst) /dev/shm/ToplingTemp/Topling-328270-K3sAqp 2024/11/22-23:13:22.669644 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233202.sst): first pass time = 0.81's, 166.193'MB/sec 2024/11/22-23:13:24.258657 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233202.sst): old prealloc_size = 147639500, real_size = 32776160 2024/11/22-23:13:24.259217 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233202.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924597 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0308 GB avg-val = 117.16 avg-zval = 33.30 2024/11/22-23:13:24.260026 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233202.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:25.184897 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233202.sst): second pass time = 1.59's, 68.205'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 39.805'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3245.910'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.291'MB/sec write SST data time = 0.13's, 281.881'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32907.613'MB/sec zip my value time = 1.59's, unzip length = 0.108'GB zip my value throughput = 68.205'MB/sec zip pipeline throughput = 131.991'MB/sec entries = 924597 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.16 avg-zval = 33.30 usrkeys = 924597 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.16 avg-zval = 33.30 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924597 bytes = 3236112 } seq expand size = 6472179 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5125 value = 3.5188 dict = 1.00 all = 3.6405 } Zip/UnZip{ index = 0.0951 value = 0.2842 dict = 1.00 all = 0.2747 } ---------------------------- total value len = 0.688760 GB avg = 0.124 KB (by entry num) total key len = 0.155396 GB avg = 0.028 KB total ukey len = 0.110997 GB avg = 0.020 KB total ukey num = 0.005549862 Billion total entry num = 0.005549862 Billion write speed all = 0.016339946 MB/sec (with seq num) write speed all = 0.015480536 MB/sec (without seq num) 2024/11/22-23:13:25.240509 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233202: 924597 keys, 36868008 bytes, temperature: kWarm 2024/11/22-23:13:25.240576 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288405240544, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233202, "file_size": 36868008, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038106, "largest_seqno": 3634540026, "table_properties": {"data_size": 30785808, "index_size": 1759040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25888716, "raw_average_key_size": 28, "raw_value_size": 108329078, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924597, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288401, "oldest_key_time": 0, "file_creation_time": 1732288401, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233202, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:25.276404 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233203.sst) /dev/shm/ToplingTemp/Topling-328270-u2OVYp 2024/11/22-23:13:26.095854 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233203.sst): first pass time = 0.82's, 163.797'MB/sec 2024/11/22-23:13:27.669527 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233203.sst): old prealloc_size = 147639500, real_size = 32461312 2024/11/22-23:13:27.670130 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233203.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929441 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0305 GB avg-val = 116.41 avg-zval = 32.77 2024/11/22-23:13:27.670907 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233203.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:28.601092 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233203.sst): second pass time = 1.57's, 68.809'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 39.129'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3036.490'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 284.301'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35872.216'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 68.809'MB/sec zip pipeline throughput = 118.713'MB/sec entries = 929441 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.41 avg-zval = 32.77 usrkeys = 929441 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.41 avg-zval = 32.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929441 bytes = 3136880 } seq expand size = 6506087 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0305 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.5148 value = 3.5519 dict = 1.00 all = 3.6822 } Zip/UnZip{ index = 0.0951 value = 0.2815 dict = 1.00 all = 0.2716 } ---------------------------- total value len = 0.803459 GB avg = 0.124 KB (by entry num) total key len = 0.181420 GB avg = 0.028 KB total ukey len = 0.129586 GB avg = 0.020 KB total ukey num = 0.006479303 Billion total entry num = 0.006479303 Billion write speed all = 0.019062613 MB/sec (with seq num) write speed all = 0.018059344 MB/sec (without seq num) 2024/11/22-23:13:28.635571 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233203: 929441 keys, 36450772 bytes, temperature: kWarm 2024/11/22-23:13:28.635635 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288408635602, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233203, "file_size": 36450772, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3522904895, "largest_seqno": 3634538655, "table_properties": {"data_size": 30460912, "index_size": 1767872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26024348, "raw_average_key_size": 28, "raw_value_size": 108193386, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929441, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288405, "oldest_key_time": 0, "file_creation_time": 1732288405, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233203, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:28.672208 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233204.sst) /dev/shm/ToplingTemp/Topling-328270-fkvbtt 2024/11/22-23:13:29.496051 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233204.sst): first pass time = 0.82's, 162.925'MB/sec 2024/11/22-23:13:31.043439 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233204.sst): old prealloc_size = 147639500, real_size = 31616240 2024/11/22-23:13:31.043993 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233204.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923309 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0296 GB avg-val = 117.37 avg-zval = 32.09 2024/11/22-23:13:31.044766 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233204.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:31.910823 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233204.sst): second pass time = 1.55's, 70.064'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 41.166'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3272.314'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.312'MB/sec write SST data time = 0.13's, 279.468'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 35702.971'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 70.064'MB/sec zip pipeline throughput = 121.126'MB/sec entries = 923309 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.37 avg-zval = 32.09 usrkeys = 923309 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.37 avg-zval = 32.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923309 bytes = 3231600 } seq expand size = 6463163 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0296 GB dict = 1.08 MB all = 0.0357 GB } UnZip/Zip{ index = 10.5132 value = 3.6574 dict = 1.00 all = 3.7590 } Zip/UnZip{ index = 0.0951 value = 0.2734 dict = 1.00 all = 0.2660 } ---------------------------- total value len = 0.918287 GB avg = 0.124 KB (by entry num) total key len = 0.207273 GB avg = 0.028 KB total ukey len = 0.148052 GB avg = 0.020 KB total ukey num = 0.007402612 Billion total entry num = 0.007402612 Billion write speed all = 0.021784136 MB/sec (with seq num) write speed all = 0.020637973 MB/sec (without seq num) 2024/11/22-23:13:31.943131 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233204: 923309 keys, 35705628 bytes, temperature: kWarm 2024/11/22-23:13:31.943197 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288411943164, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233204, "file_size": 35705628, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038136, "largest_seqno": 3634539876, "table_properties": {"data_size": 29628768, "index_size": 1756480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25852652, "raw_average_key_size": 28, "raw_value_size": 108365114, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923309, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288408, "oldest_key_time": 0, "file_creation_time": 1732288408, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233204, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:31.980338 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233205.sst) /dev/shm/ToplingTemp/Topling-328270-NKWiCs 2024/11/22-23:13:32.808168 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233205.sst): first pass time = 0.83's, 162.140'MB/sec 2024/11/22-23:13:34.303568 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233205.sst): old prealloc_size = 147639500, real_size = 31900576 2024/11/22-23:13:34.304130 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233205.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926579 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0299 GB avg-val = 116.85 avg-zval = 32.28 2024/11/22-23:13:34.304917 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233205.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:35.189201 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233205.sst): second pass time = 1.49's, 72.440'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 40.647'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3274.377'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.306'MB/sec write SST data time = 0.13's, 282.163'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33602.613'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.440'MB/sec zip pipeline throughput = 99.088'MB/sec entries = 926579 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 32.28 usrkeys = 926579 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 32.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926579 bytes = 3243040 } seq expand size = 6486053 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0299 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.5117 value = 3.6205 dict = 1.00 all = 3.7285 } Zip/UnZip{ index = 0.0951 value = 0.2762 dict = 1.00 all = 0.2682 } ---------------------------- total value len = 1.033047 GB avg = 0.124 KB (by entry num) total key len = 0.233217 GB avg = 0.028 KB total ukey len = 0.166584 GB avg = 0.020 KB total ukey num = 0.008329191 Billion total entry num = 0.008329191 Billion write speed all = 0.024505769 MB/sec (with seq num) write speed all = 0.023216224 MB/sec (without seq num) 2024/11/22-23:13:35.221234 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233205: 926579 keys, 35998100 bytes, temperature: kWarm 2024/11/22-23:13:35.221298 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288415221265, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233205, "file_size": 35998100, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3499102480, "largest_seqno": 3634539778, "table_properties": {"data_size": 29905808, "index_size": 1762944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944212, "raw_average_key_size": 28, "raw_value_size": 108273654, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926579, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288411, "oldest_key_time": 0, "file_creation_time": 1732288411, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233205, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:35.259051 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233206.sst) /dev/shm/ToplingTemp/Topling-328270-xyjhWo 2024/11/22-23:13:36.074197 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233206.sst): first pass time = 0.82's, 164.664'MB/sec 2024/11/22-23:13:37.630201 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233206.sst): old prealloc_size = 147639500, real_size = 37221536 2024/11/22-23:13:37.630800 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233206.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926206 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0352 GB avg-val = 116.91 avg-zval = 38.02 2024/11/22-23:13:37.631281 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233206.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:38.364694 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233206.sst): second pass time = 1.55's, 69.642'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.73's, 54.382'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3061.476'MB/sec (index lex order gen) rebuild zvType time = 0.61's, 0.382'MB/sec write SST data time = 0.13's, 311.879'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30942.885'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 69.642'MB/sec zip pipeline throughput = 69.642'MB/sec entries = 926206 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.91 avg-zval = 38.02 usrkeys = 926206 avg-key = 20.00 avg-zkey = 1.91 avg-val = 120.49 avg-zval = 38.02 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 473881 bytes = 1717840 } seq expand size = 3317167 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0352 GB dict = 1.08 MB all = 0.0399 GB } UnZip/Zip{ index = 10.4449 value = 3.0748 dict = 1.00 all = 3.3603 } Zip/UnZip{ index = 0.0957 value = 0.3252 dict = 1.00 all = 0.2976 } ---------------------------- total value len = 1.144648 GB avg = 0.124 KB (by entry num) total key len = 0.259151 GB avg = 0.028 KB total ukey len = 0.185108 GB avg = 0.020 KB total ukey num = 0.009255397 Billion total entry num = 0.009255397 Billion write speed all = 0.027165787 MB/sec (with seq num) write speed all = 0.025732931 MB/sec (without seq num) 2024/11/22-23:13:38.401591 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233206: 926206 keys, 39942309 bytes, temperature: kWarm 2024/11/22-23:13:38.401659 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288418401628, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233206, "file_size": 39942309, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3634536129, "table_properties": {"data_size": 35216272, "index_size": 1773504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25933768, "raw_average_key_size": 28, "raw_value_size": 108283988, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926206, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288415, "oldest_key_time": 0, "file_creation_time": 1732288415, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233206, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:38.443961 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233207.sst) /dev/shm/ToplingTemp/Topling-328270-CSgc9s 2024/11/22-23:13:39.201933 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233207.sst): first pass time = 0.76's, 177.083'MB/sec 2024/11/22-23:13:40.933318 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233207.sst): old prealloc_size = 147639500, real_size = 36440256 2024/11/22-23:13:40.933907 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233207.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926902 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.80 avg-zval = 37.17 2024/11/22-23:13:40.933951 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233207.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:41.573991 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233207.sst): second pass time = 1.73's, 62.556'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.64's, 58.218'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3086.433'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.450'MB/sec write SST data time = 0.12's, 298.487'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33680.160'MB/sec zip my value time = 1.73's, unzip length = 0.108'GB zip my value throughput = 62.556'MB/sec zip pipeline throughput = 62.556'MB/sec entries = 926902 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 37.17 usrkeys = 926902 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 37.17 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5818 value = 3.1421 dict = 1.00 all = 3.5985 } Zip/UnZip{ index = 0.0945 value = 0.3183 dict = 1.00 all = 0.2779 } ---------------------------- total value len = 1.252913 GB avg = 0.123 KB (by entry num) total key len = 0.285104 GB avg = 0.028 KB total ukey len = 0.203646 GB avg = 0.020 KB total ukey num = 0.010182299 Billion total entry num = 0.010182299 Billion write speed all = 0.029761270 MB/sec (with seq num) write speed all = 0.028185016 MB/sec (without seq num) 2024/11/22-23:13:41.610078 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233207: 926902 keys, 37298038 bytes, temperature: kWarm 2024/11/22-23:13:41.610147 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288421610111, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233207, "file_size": 37298038, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34456496, "index_size": 1751872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953256, "raw_average_key_size": 28, "raw_value_size": 108264556, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926902, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288418, "oldest_key_time": 0, "file_creation_time": 1732288418, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233207, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:41.652930 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233208.sst) /dev/shm/ToplingTemp/Topling-328270-ce2n0p 2024/11/22-23:13:42.540815 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233208.sst): first pass time = 0.89's, 151.173'MB/sec 2024/11/22-23:13:44.174738 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233208.sst): old prealloc_size = 147639500, real_size = 36938736 2024/11/22-23:13:44.175299 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233208.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925980 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0349 GB avg-val = 116.95 avg-zval = 37.74 2024/11/22-23:13:44.175505 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233208.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:44.837436 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233208.sst): second pass time = 1.63's, 66.305'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.66's, 57.856'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3261.304'MB/sec (index lex order gen) rebuild zvType time = 0.54's, 0.430'MB/sec write SST data time = 0.12's, 308.239'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 38394.313'MB/sec zip my value time = 1.63's, unzip length = 0.108'GB zip my value throughput = 66.305'MB/sec zip pipeline throughput = 66.305'MB/sec entries = 925980 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.95 avg-zval = 37.74 usrkeys = 925980 avg-key = 20.00 avg-zkey = 1.90 avg-val = 117.91 avg-zval = 37.74 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 127436 bytes = 398256 } seq expand size = 892052 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0349 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5129 value = 3.0988 dict = 1.00 all = 3.5007 } Zip/UnZip{ index = 0.0951 value = 0.3227 dict = 1.00 all = 0.2857 } ---------------------------- total value len = 1.362095 GB avg = 0.123 KB (by entry num) total key len = 0.311032 GB avg = 0.028 KB total ukey len = 0.222166 GB avg = 0.020 KB total ukey num = 0.011108279 Billion total entry num = 0.011108279 Billion write speed all = 0.032373657 MB/sec (with seq num) write speed all = 0.030654167 MB/sec (without seq num) 2024/11/22-23:13:44.874604 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233208: 925980 keys, 38340079 bytes, temperature: kWarm 2024/11/22-23:13:44.874676 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288424874636, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233208, "file_size": 38340079, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3299889979, "table_properties": {"data_size": 34945440, "index_size": 1761600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25927440, "raw_average_key_size": 28, "raw_value_size": 108290360, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925980, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288421, "oldest_key_time": 0, "file_creation_time": 1732288421, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233208, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:44.908191 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233209.sst) /dev/shm/ToplingTemp/Topling-328270-ojOwYp 2024/11/22-23:13:45.762366 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233209.sst): first pass time = 0.85's, 157.139'MB/sec 2024/11/22-23:13:47.569333 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233209.sst): old prealloc_size = 147639500, real_size = 30876128 2024/11/22-23:13:47.569891 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233209.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925417 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0289 GB avg-val = 117.03 avg-zval = 31.21 2024/11/22-23:13:47.572495 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233209.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:48.569174 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233209.sst): second pass time = 1.81's, 59.961'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 35.093'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3284.558'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.304'MB/sec write SST data time = 0.24's, 147.025'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32598.570'MB/sec zip my value time = 1.81's, unzip length = 0.108'GB zip my value throughput = 59.961'MB/sec zip pipeline throughput = 132.834'MB/sec entries = 925417 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.03 avg-zval = 31.21 usrkeys = 925417 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.03 avg-zval = 31.21 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925417 bytes = 3354656 } seq expand size = 6477919 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0289 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.5119 value = 3.7497 dict = 1.00 all = 3.8254 } Zip/UnZip{ index = 0.0951 value = 0.2667 dict = 1.00 all = 0.2614 } ---------------------------- total value len = 1.476879 GB avg = 0.123 KB (by entry num) total key len = 0.336943 GB avg = 0.028 KB total ukey len = 0.240674 GB avg = 0.020 KB total ukey num = 0.012033696 Billion total entry num = 0.012033696 Billion write speed all = 0.035093473 MB/sec (with seq num) write speed all = 0.033230869 MB/sec (without seq num) 2024/11/22-23:13:48.607391 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233209: 925417 keys, 35085810 bytes, temperature: kWarm 2024/11/22-23:13:48.607471 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288428607427, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233209, "file_size": 35085810, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3299889980, "largest_seqno": 3634538620, "table_properties": {"data_size": 28883856, "index_size": 1760704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25911676, "raw_average_key_size": 28, "raw_value_size": 108306158, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925417, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288424, "oldest_key_time": 0, "file_creation_time": 1732288424, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233209, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:48.673574 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233210.sst) /dev/shm/ToplingTemp/Topling-328270-WwvYEp 2024/11/22-23:13:49.565530 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233210.sst): first pass time = 0.89's, 150.484'MB/sec 2024/11/22-23:13:51.093239 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233210.sst): old prealloc_size = 147639500, real_size = 35187904 2024/11/22-23:13:51.093795 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233210.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928224 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 116.60 avg-zval = 35.74 2024/11/22-23:13:51.094608 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233210.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:52.183146 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233210.sst): second pass time = 1.53's, 70.880'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.09's, 36.156'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3313.874'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.242'MB/sec write SST data time = 0.13's, 303.528'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35695.715'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.880'MB/sec zip pipeline throughput = 70.880'MB/sec entries = 928224 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.60 avg-zval = 35.74 usrkeys = 928224 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.60 avg-zval = 35.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928224 bytes = 3364832 } seq expand size = 6497568 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.4417 value = 3.2621 dict = 1.00 all = 3.4060 } Zip/UnZip{ index = 0.0958 value = 0.3066 dict = 1.00 all = 0.2936 } ---------------------------- total value len = 1.591604 GB avg = 0.123 KB (by entry num) total key len = 0.362934 GB avg = 0.028 KB total ukey len = 0.259238 GB avg = 0.020 KB total ukey num = 0.012961920 Billion total entry num = 0.012961920 Billion write speed all = 0.037813362 MB/sec (with seq num) write speed all = 0.035807226 MB/sec (without seq num) 2024/11/22-23:13:52.220412 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233210: 928224 keys, 39406204 bytes, temperature: kWarm 2024/11/22-23:13:52.220480 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288432220448, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233210, "file_size": 39406204, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701386, "largest_seqno": 3634539994, "table_properties": {"data_size": 33177776, "index_size": 1777920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25990272, "raw_average_key_size": 28, "raw_value_size": 108227584, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928224, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288428, "oldest_key_time": 0, "file_creation_time": 1732288428, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233210, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:52.256824 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233211.sst) /dev/shm/ToplingTemp/Topling-328270-PUN0is 2024/11/22-23:13:53.392473 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233211.sst): first pass time = 1.14's, 118.190'MB/sec 2024/11/22-23:13:54.992190 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233211.sst): old prealloc_size = 147639500, real_size = 33698080 2024/11/22-23:13:54.992746 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233211.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930254 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0317 GB avg-val = 116.28 avg-zval = 34.06 2024/11/22-23:13:54.995252 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233211.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:13:55.915507 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233211.sst): second pass time = 1.60's, 67.650'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 40.947'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3336.480'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 291.963'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33974.742'MB/sec zip my value time = 1.60's, unzip length = 0.108'GB zip my value throughput = 67.650'MB/sec zip pipeline throughput = 112.202'MB/sec entries = 930254 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.28 avg-zval = 34.06 usrkeys = 930254 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.28 avg-zval = 34.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930254 bytes = 3255904 } seq expand size = 6511778 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.4480 value = 3.4140 dict = 1.00 all = 3.5502 } Zip/UnZip{ index = 0.0957 value = 0.2929 dict = 1.00 all = 0.2817 } ---------------------------- total value len = 1.706287 GB avg = 0.123 KB (by entry num) total key len = 0.388981 GB avg = 0.028 KB total ukey len = 0.277843 GB avg = 0.020 KB total ukey num = 0.013892174 Billion total entry num = 0.013892174 Billion write speed all = 0.040533052 MB/sec (with seq num) write speed all = 0.038383094 MB/sec (without seq num) 2024/11/22-23:13:55.951633 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233211: 930254 keys, 37806078 bytes, temperature: kWarm 2024/11/22-23:13:55.951695 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288435951663, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233211, "file_size": 37806078, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3634539835, "table_properties": {"data_size": 31684624, "index_size": 1780736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26047112, "raw_average_key_size": 28, "raw_value_size": 108170680, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930254, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288432, "oldest_key_time": 0, "file_creation_time": 1732288432, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233211, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:55.992556 140024986822208 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233212.sst) /dev/shm/ToplingTemp/Topling-328270-P7jKXs 2024/11/22-23:13:56.083986 140024986822208 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233212.sst): first pass time = 0.09's, 238.395'MB/sec 2024/11/22-23:13:56.266922 140024986822208 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233212.sst): old prealloc_size = 147639500, real_size = 6464304 2024/11/22-23:13:56.267061 140024986822208 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233212.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 150745 prefix = 4 raw-key = 0.0024 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0176 GB zip-val = 0.0061 GB avg-val = 116.53 avg-zval = 40.71 2024/11/22-23:13:56.267210 140024986822208 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233212.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.022 GB 2024/11/22-23:13:56.402173 140024986822208 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00/233212.sst): second pass time = 0.18's, 96.305'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.14's, 52.660'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2564.888'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.331'MB/sec write SST data time = 0.02's, 333.678'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.184'MB, throughput = 4952.179'MB/sec zip my value time = 0.18's, unzip length = 0.018'GB zip my value throughput = 96.305'MB/sec zip pipeline throughput = 96.305'MB/sec entries = 150745 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.53 avg-zval = 40.71 usrkeys = 150745 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.53 avg-zval = 40.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 150745 bytes = 508784 } seq expand size = 1055215 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0030 GB value = 0.0176 GB dict = 0.18 MB all = 0.0218 GB } __ZipSize{ index = 0.0003 GB value = 0.0061 GB dict = 0.18 MB all = 0.0071 GB } UnZip/Zip{ index = 10.3899 value = 2.8627 dict = 1.00 all = 3.0592 } Zip/UnZip{ index = 0.0962 value = 0.3493 dict = 1.00 all = 0.3269 } ---------------------------- total value len = 1.724908 GB avg = 0.123 KB (by entry num) total key len = 0.393202 GB avg = 0.028 KB total ukey len = 0.280858 GB avg = 0.020 KB total ukey num = 0.014042919 Billion total entry num = 0.014042919 Billion write speed all = 0.040974549 MB/sec (with seq num) write speed all = 0.038801282 MB/sec (without seq num) 2024/11/22-23:13:56.415517 140024986822208 [compaction_job.cc:1948] [default] [JOB 13679] Generated table #233212: 150745 keys, 7121689 bytes, temperature: kWarm 2024/11/22-23:13:56.415574 140024986822208 EVENT_LOG_v1 {"time_micros": 1732288436415544, "cf_name": "default", "job": 13679, "event": "table_file_creation", "file_number": 233212, "file_size": 7121689, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3563652023, "largest_seqno": 3633073393, "table_properties": {"data_size": 6136288, "index_size": 290176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4220860, "raw_average_key_size": 28, "raw_value_size": 17566074, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 150745, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732288435, "oldest_key_time": 0, "file_creation_time": 1732288435, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 233212, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:13:56.428493 140024986822208 [compaction_job.cc:703] job-13679: subcompact[0], size: 0.565496 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-13679/att-00 [233197,233198,233199,233200,233201,233202,233203,233204,233205,233206,233207,233208,233209,233210,233211,233212] 2024/11/22-23:13:56.529279 140024986822208 (Original Log Time 2024/11/22-23:13:02.205665) EVENT_LOG_v1 {"time_micros": 1732288382205625, "job": 13679, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [233192], "files_L3": [233141, 233142, 233143, 233144, 233145, 233146, 233147, 233148, 233149, 233150, 233151, 233152, 233153, 233154, 233156], "score": 0.998756, "input_data_size": 585859138, "oldest_snapshot_seqno": 3634561182} 2024/11/22-23:13:56.529282 140024986822208 (Original Log Time 2024/11/22-23:13:56.528467) [compaction_job.cc:2022] [default] [JOB 13679] Compacted 1@2 + 15@3 files to L3 => 565496017 bytes 2024/11/22-23:13:56.529283 140024986822208 (Original Log Time 2024/11/22-23:13:56.529216) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 16 0 0 0] max score 0.02, MB/sec: 10.8 rd, 10.4 wr, level 3, files in(1, 15) out(16 +0 blob) MB in(27.1, 531.6 +0.0 blob) out(539.3 +0.0 blob), read-write-amplify(40.5) write-amplify(19.9) OK, records in: 14125490, records dropped: 82571 output_compression: Snappy 2024/11/22-23:13:56.529288 140024986822208 (Original Log Time 2024/11/22-23:13:56.529260) EVENT_LOG_v1 {"time_micros": 1732288436529239, "cf": "default", "job": 13679, "event": "compaction_finished", "compaction_time_micros": 54222834, "compaction_time_cpu_micros": 22696996, "output_level": 3, "num_output_files": 16, "total_output_size": 565496017, "num_input_records": 14125490, "num_output_records": 14042919, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 390927398, "file_range_sync_nanos": 0, "file_fsync_nanos": 16287, "file_prepare_write_nanos": 36670, "lsm_state": [0, 0, 0, 16, 0, 0, 0]} 2024/11/22-23:13:56.679430 140024986822208 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732288382,"executesMs":54452,"compactionJobId":13679,"attempt":0,"compactionInputRawBytes":2046778928,"compactionInputZipBytes":585859138,"compactionOutputRawBytes":0,"compactionOutputZipBytes":565496017,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-9b83e8b919b27dd9ff1738cb49e0330a-18e55c27475b075a-00"}}, response = 2024/11/22-23:13:56.679493 140024986822208 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-13679/att-00: olev 3, work 54.452 s, result 6.310 ms, install 0.840 ms, input{raw 1.906 GiB zip 558.719 MiB}