2024/11/22-17:52:40.677583 140024777004608 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00: fork to child time = 0.025566 sec 2024/11/22-17:52:40.678611 140024777004608 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00 2024/11/22-17:52:40.688162 140024777004608 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00 2024/11/22-17:52:40.689651 140024777004608 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-17:52:40.690016 140024777004608 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-17:52:40.734423 140024777004608 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-06645/att-00/MANIFEST-206347 2024/11/22-17:52:40.748592 140024777004608 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-06645/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 218766, last_sequence is 3440144469, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 218630 2024/11/22-17:52:40.748604 140024777004608 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-17:52:40.748745 140024777004608 [version_set.cc:5721] Creating manifest 218766 2024/11/22-17:52:40.780413 140024777004608 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-17:52:40.800939 140024777004608 [compaction_job.cc:2437] [default] [JOB 6645] Compacting 1@2 + 13@3 files to L3, score 0.95, subcompactions 1 : 1 2024/11/22-17:52:40.800964 140024777004608 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 5088 Base level 2, inputs: [218762(27MB)], [218725(36MB) 218726(34MB) 218727(37MB) 218728(36MB) 218729(35MB) 218730(11MB) 218707(36MB) 218708(36MB) 218709(36MB) 218710(35MB) 218711(36MB) 218712(36MB) 218713(36MB)] 2024/11/22-17:52:40.809686 140024777004608 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-17:52:40.809731 140024777004608 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-17:52:40.809758 140024777004608 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-17:52:40.809767 140024777004608 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-17:52:40.809793 140024777004608 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-17:52:41.425226 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218767.sst) /dev/shm/ToplingTemp/Topling-228151-iCCsdV 2024/11/22-17:52:42.490288 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218767.sst): first pass time = 1.07's, 126.024'MB/sec 2024/11/22-17:52:44.167108 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218767.sst): old prealloc_size = 147639500, real_size = 33993216 2024/11/22-17:52:44.167824 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218767.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927907 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0320 GB avg-val = 116.65 avg-zval = 34.47 2024/11/22-17:52:44.168657 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218767.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:52:45.392228 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218767.sst): second pass time = 1.68's, 64.583'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.23's, 31.006'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2549.706'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.275'MB/sec write SST data time = 0.38's, 100.004'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 37864.114'MB/sec zip my value time = 1.68's, unzip length = 0.108'GB zip my value throughput = 64.583'MB/sec zip pipeline throughput = 65.616'MB/sec entries = 927907 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.65 avg-zval = 34.47 usrkeys = 927907 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.58 avg-zval = 34.47 TagRS{ kinds = RS_Key0_TagN bytes = 145080 } TagArray{ size = 919837 bytes = 2989488 } seq expand size = 6438859 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.4453 value = 3.3840 dict = 1.00 all = 3.5334 } Zip/UnZip{ index = 0.0957 value = 0.2955 dict = 1.00 all = 0.2830 } ---------------------------- total value len = 0.114675 GB avg = 0.124 KB (by entry num) total key len = 0.025981 GB avg = 0.028 KB total ukey len = 0.018558 GB avg = 0.020 KB total ukey num = 0.000927907 Billion total entry num = 0.000927907 Billion write speed all = 0.004338268 MB/sec (with seq num) write speed all = 0.004109313 MB/sec (without seq num) 2024/11/22-17:52:45.429949 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218767: 927907 keys, 37985396 bytes, temperature: kWarm 2024/11/22-17:52:45.430055 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269165429987, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218767, "file_size": 37985396, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3440102239, "table_properties": {"data_size": 31984368, "index_size": 1776704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25981396, "raw_average_key_size": 28, "raw_value_size": 108236362, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927907, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269161, "oldest_key_time": 0, "file_creation_time": 1732269161, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218767, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:52:45.506264 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218768.sst) /dev/shm/ToplingTemp/Topling-228151-8XbnnT 2024/11/22-17:52:47.191021 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218768.sst): first pass time = 1.68's, 79.669'MB/sec 2024/11/22-17:52:49.087649 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218768.sst): old prealloc_size = 147639500, real_size = 34990752 2024/11/22-17:52:49.088461 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218768.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1081552 prefix = 4 raw-key = 0.0173 GB zip-key = 0.0021 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1039 GB zip-val = 0.0326 GB avg-val = 96.10 avg-zval = 30.18 2024/11/22-17:52:49.088514 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218768.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0028 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:52:50.197992 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218768.sst): second pass time = 1.90's, 54.822'MB/sec, value only(77.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.11's, 32.208'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2649.316'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.325'MB/sec write SST data time = 0.28's, 128.521'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.029'MB, throughput = 24995.045'MB/sec zip my value time = 1.90's, unzip length = 0.104'GB zip my value throughput = 54.822'MB/sec zip pipeline throughput = 54.822'MB/sec entries = 1081552 avg-key = 28.00 avg-zkey = 1.93 avg-val = 96.10 avg-zval = 30.18 usrkeys = 1081552 avg-key = 20.00 avg-zkey = 1.93 avg-val = 96.10 avg-zval = 30.18 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0216 GB value = 0.1039 GB dict = 1.03 MB all = 0.1342 GB } __ZipSize{ index = 0.0021 GB value = 0.0326 GB dict = 1.03 MB all = 0.0358 GB } UnZip/Zip{ index = 10.3797 value = 3.1846 dict = 1.00 all = 3.7533 } Zip/UnZip{ index = 0.0963 value = 0.3140 dict = 1.00 all = 0.2664 } ---------------------------- total value len = 0.218610 GB avg = 0.109 KB (by entry num) total key len = 0.056265 GB avg = 0.028 KB total ukey len = 0.040189 GB avg = 0.020 KB total ukey num = 0.002009459 Billion total entry num = 0.002009459 Billion write speed all = 0.008476689 MB/sec (with seq num) write speed all = 0.007980941 MB/sec (without seq num) 2024/11/22-17:52:50.231888 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218768: 1081552 keys, 35760426 bytes, temperature: kWarm 2024/11/22-17:52:50.231954 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269170231917, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218768, "file_size": 35760426, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 32636240, "index_size": 2083968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 30283456, "raw_average_key_size": 28, "raw_value_size": 103934397, "raw_average_value_size": 96, "num_data_blocks": 1, "num_entries": 1081552, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269165, "oldest_key_time": 0, "file_creation_time": 1732269165, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218768, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:52:50.271287 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218769.sst) /dev/shm/ToplingTemp/Topling-228151-tYE6mS 2024/11/22-17:52:52.049910 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218769.sst): first pass time = 1.78's, 75.463'MB/sec 2024/11/22-17:52:54.282011 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218769.sst): old prealloc_size = 147639500, real_size = 35651600 2024/11/22-17:52:54.282579 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218769.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924778 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0337 GB avg-val = 117.14 avg-zval = 36.40 2024/11/22-17:52:54.283199 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218769.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:52:55.259933 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218769.sst): second pass time = 2.23's, 48.558'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 39.725'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3196.326'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.311'MB/sec write SST data time = 0.23's, 165.465'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32389.405'MB/sec zip my value time = 2.23's, unzip length = 0.108'GB zip my value throughput = 48.558'MB/sec zip pipeline throughput = 57.244'MB/sec entries = 924778 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.14 avg-zval = 36.40 usrkeys = 924778 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.87 avg-zval = 36.40 TagRS{ kinds = RS_Key0_TagN bytes = 144600 } TagArray{ size = 625997 bytes = 2191008 } seq expand size = 4381979 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5127 value = 3.2181 dict = 1.00 all = 3.4551 } Zip/UnZip{ index = 0.0951 value = 0.3107 dict = 1.00 all = 0.2894 } ---------------------------- total value len = 0.331316 GB avg = 0.113 KB (by entry num) total key len = 0.082159 GB avg = 0.028 KB total ukey len = 0.058685 GB avg = 0.020 KB total ukey num = 0.002934237 Billion total entry num = 0.002934237 Billion write speed all = 0.012748896 MB/sec (with seq num) write speed all = 0.012025112 MB/sec (without seq num) 2024/11/22-17:52:55.303376 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218769: 924778 keys, 38846548 bytes, temperature: kWarm 2024/11/22-17:52:55.303478 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269175303434, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218769, "file_size": 38846548, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3440102481, "table_properties": {"data_size": 33660864, "index_size": 1759360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25893784, "raw_average_key_size": 28, "raw_value_size": 108324060, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924778, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269170, "oldest_key_time": 0, "file_creation_time": 1732269170, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218769, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:52:55.381440 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218770.sst) /dev/shm/ToplingTemp/Topling-228151-Nq04NV 2024/11/22-17:52:56.661419 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218770.sst): first pass time = 1.28's, 104.863'MB/sec 2024/11/22-17:52:58.563305 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218770.sst): old prealloc_size = 147639500, real_size = 34569696 2024/11/22-17:52:58.563973 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218770.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925143 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0326 GB avg-val = 117.08 avg-zval = 35.21 2024/11/22-17:52:58.564829 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218770.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:53:00.058681 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218770.sst): second pass time = 1.90's, 57.031'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.50's, 25.855'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2715.967'MB/sec (index lex order gen) rebuild zvType time = 1.22's, 0.189'MB/sec write SST data time = 0.27's, 142.163'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32161.076'MB/sec zip my value time = 1.90's, unzip length = 0.108'GB zip my value throughput = 57.031'MB/sec zip pipeline throughput = 74.327'MB/sec entries = 925143 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.08 avg-zval = 35.21 usrkeys = 925143 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.08 avg-zval = 35.21 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925143 bytes = 3238016 } seq expand size = 6476001 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5107 value = 3.3248 dict = 1.00 all = 3.4715 } Zip/UnZip{ index = 0.0951 value = 0.3008 dict = 1.00 all = 0.2881 } ---------------------------- total value len = 0.446105 GB avg = 0.116 KB (by entry num) total key len = 0.108063 GB avg = 0.028 KB total ukey len = 0.077188 GB avg = 0.020 KB total ukey num = 0.003859380 Billion total entry num = 0.003859380 Billion write speed all = 0.017084462 MB/sec (with seq num) write speed all = 0.016132615 MB/sec (without seq num) 2024/11/22-17:53:00.119192 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218770: 925143 keys, 38662868 bytes, temperature: kWarm 2024/11/22-17:53:00.119274 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269180119230, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218770, "file_size": 38662868, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3440102732, "table_properties": {"data_size": 32577872, "index_size": 1760384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25904004, "raw_average_key_size": 28, "raw_value_size": 108313762, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925143, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269175, "oldest_key_time": 0, "file_creation_time": 1732269175, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218770, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:53:00.189752 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218771.sst) /dev/shm/ToplingTemp/Topling-228151-sIJaNS 2024/11/22-17:53:01.321403 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218771.sst): first pass time = 1.13's, 118.610'MB/sec 2024/11/22-17:53:02.964056 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218771.sst): old prealloc_size = 147639500, real_size = 33305968 2024/11/22-17:53:02.964740 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218771.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922328 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0313 GB avg-val = 117.52 avg-zval = 33.95 2024/11/22-17:53:02.965578 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218771.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:53:04.332117 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218771.sst): second pass time = 1.64's, 66.056'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.37's, 27.333'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2677.434'MB/sec (index lex order gen) rebuild zvType time = 1.10's, 0.209'MB/sec write SST data time = 0.26's, 142.027'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 35562.857'MB/sec zip my value time = 1.64's, unzip length = 0.108'GB zip my value throughput = 66.056'MB/sec zip pipeline throughput = 66.056'MB/sec entries = 922328 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.52 avg-zval = 33.95 usrkeys = 922328 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.52 avg-zval = 33.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922328 bytes = 3228160 } seq expand size = 6456296 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.4426 value = 3.4621 dict = 1.00 all = 3.5894 } Zip/UnZip{ index = 0.0958 value = 0.2888 dict = 1.00 all = 0.2786 } ---------------------------- total value len = 0.560954 GB avg = 0.117 KB (by entry num) total key len = 0.133888 GB avg = 0.028 KB total ukey len = 0.095634 GB avg = 0.020 KB total ukey num = 0.004781708 Billion total entry num = 0.004781708 Billion write speed all = 0.021418488 MB/sec (with seq num) write speed all = 0.020239320 MB/sec (without seq num) 2024/11/22-17:53:04.369972 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218771: 922328 keys, 37392978 bytes, temperature: kWarm 2024/11/22-17:53:04.370071 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269184370008, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218771, "file_size": 37392978, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701122, "largest_seqno": 3440102644, "table_properties": {"data_size": 31308768, "index_size": 1766464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25825184, "raw_average_key_size": 28, "raw_value_size": 108392672, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922328, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269180, "oldest_key_time": 0, "file_creation_time": 1732269180, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218771, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:53:04.428292 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218772.sst) /dev/shm/ToplingTemp/Topling-228151-sVrD7R 2024/11/22-17:53:05.728938 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218772.sst): first pass time = 1.30's, 103.197'MB/sec 2024/11/22-17:53:07.201656 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218772.sst): old prealloc_size = 147639500, real_size = 34553888 2024/11/22-17:53:07.202248 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218772.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925162 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0325 GB avg-val = 117.07 avg-zval = 35.17 2024/11/22-17:53:07.203004 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218772.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:53:08.116698 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218772.sst): second pass time = 1.47's, 73.577'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 42.110'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3129.750'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.294'MB/sec write SST data time = 0.13's, 299.257'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33991.763'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.577'MB/sec zip pipeline throughput = 73.577'MB/sec entries = 925162 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.07 avg-zval = 35.17 usrkeys = 925162 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.07 avg-zval = 35.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925162 bytes = 3122432 } seq expand size = 6476134 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.3755 value = 3.3287 dict = 1.00 all = 3.4833 } Zip/UnZip{ index = 0.0964 value = 0.3004 dict = 1.00 all = 0.2871 } ---------------------------- total value len = 0.675744 GB avg = 0.118 KB (by entry num) total key len = 0.159792 GB avg = 0.028 KB total ukey len = 0.114137 GB avg = 0.020 KB total ukey num = 0.005706870 Billion total entry num = 0.005706870 Billion write speed all = 0.025752370 MB/sec (with seq num) write speed all = 0.024345221 MB/sec (without seq num) 2024/11/22-17:53:08.151772 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218772: 925162 keys, 38532008 bytes, temperature: kWarm 2024/11/22-17:53:08.151873 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269188151840, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218772, "file_size": 38532008, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3326330223, "largest_seqno": 3440102558, "table_properties": {"data_size": 32539024, "index_size": 1783360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25904536, "raw_average_key_size": 28, "raw_value_size": 108313288, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925162, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269184, "oldest_key_time": 0, "file_creation_time": 1732269184, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218772, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:53:08.192933 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218773.sst) /dev/shm/ToplingTemp/Topling-228151-JZDwsW 2024/11/22-17:53:09.028728 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218773.sst): first pass time = 0.84's, 160.597'MB/sec 2024/11/22-17:53:10.808194 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218773.sst): old prealloc_size = 147639500, real_size = 33542864 2024/11/22-17:53:10.808865 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218773.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924729 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0316 GB avg-val = 117.14 avg-zval = 34.12 2024/11/22-17:53:10.809688 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218773.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:53:11.974043 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218773.sst): second pass time = 1.78's, 60.902'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.17's, 32.282'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2690.584'MB/sec (index lex order gen) rebuild zvType time = 1.01's, 0.229'MB/sec write SST data time = 0.16's, 242.650'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 36363.294'MB/sec zip my value time = 1.78's, unzip length = 0.108'GB zip my value throughput = 60.902'MB/sec zip pipeline throughput = 60.902'MB/sec entries = 924729 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.14 avg-zval = 34.12 usrkeys = 924729 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.14 avg-zval = 34.12 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924729 bytes = 3236560 } seq expand size = 6473103 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5117 value = 3.4332 dict = 1.00 all = 3.5663 } Zip/UnZip{ index = 0.0951 value = 0.2913 dict = 1.00 all = 0.2804 } ---------------------------- total value len = 0.790542 GB avg = 0.119 KB (by entry num) total key len = 0.185685 GB avg = 0.028 KB total ukey len = 0.132632 GB avg = 0.020 KB total ukey num = 0.006631599 Billion total entry num = 0.006631599 Billion write speed all = 0.030085078 MB/sec (with seq num) write speed all = 0.028450113 MB/sec (without seq num) 2024/11/22-17:53:12.010618 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218773: 924729 keys, 37635472 bytes, temperature: kWarm 2024/11/22-17:53:12.010683 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269192010652, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218773, "file_size": 37635472, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701165, "largest_seqno": 3440102595, "table_properties": {"data_size": 31552064, "index_size": 1759424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25892412, "raw_average_key_size": 28, "raw_value_size": 108325338, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924729, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269188, "oldest_key_time": 0, "file_creation_time": 1732269188, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218773, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:53:12.046212 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218774.sst) /dev/shm/ToplingTemp/Topling-228151-7p6GAW 2024/11/22-17:53:13.094624 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218774.sst): first pass time = 1.05's, 128.025'MB/sec 2024/11/22-17:53:14.447441 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218774.sst): old prealloc_size = 147639500, real_size = 33902400 2024/11/22-17:53:14.448027 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218774.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926332 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0319 GB avg-val = 116.89 avg-zval = 34.46 2024/11/22-17:53:14.448800 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218774.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:53:15.373039 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218774.sst): second pass time = 1.35's, 80.081'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.899'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3098.846'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.292'MB/sec write SST data time = 0.13's, 285.738'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34579.935'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.081'MB/sec zip pipeline throughput = 80.081'MB/sec entries = 926332 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.89 avg-zval = 34.46 usrkeys = 926332 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.50 avg-zval = 34.46 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 874072 bytes = 2950000 } seq expand size = 6118504 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.5815 value = 3.3923 dict = 1.00 all = 3.5455 } Zip/UnZip{ index = 0.0945 value = 0.2948 dict = 1.00 all = 0.2820 } ---------------------------- total value len = 0.904941 GB avg = 0.120 KB (by entry num) total key len = 0.211622 GB avg = 0.028 KB total ukey len = 0.151159 GB avg = 0.020 KB total ukey num = 0.007557931 Billion total entry num = 0.007557931 Billion write speed all = 0.034406315 MB/sec (with seq num) write speed all = 0.032543165 MB/sec (without seq num) 2024/11/22-17:53:15.405699 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218774: 926332 keys, 37855456 bytes, temperature: kWarm 2024/11/22-17:53:15.405762 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269195405729, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218774, "file_size": 37855456, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3440102815, "table_properties": {"data_size": 31919792, "index_size": 1750848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25937296, "raw_average_key_size": 28, "raw_value_size": 108280444, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926332, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269192, "oldest_key_time": 0, "file_creation_time": 1732269192, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218774, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:53:15.441132 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218775.sst) /dev/shm/ToplingTemp/Topling-228151-VQVGVV 2024/11/22-17:53:16.319047 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218775.sst): first pass time = 0.88's, 152.890'MB/sec 2024/11/22-17:53:17.853447 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218775.sst): old prealloc_size = 147639500, real_size = 36544208 2024/11/22-17:53:17.854010 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218775.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924598 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 117.16 avg-zval = 37.37 2024/11/22-17:53:17.854328 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218775.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:53:18.572860 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218775.sst): second pass time = 1.53's, 70.651'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.72's, 53.472'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3246.198'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.391'MB/sec write SST data time = 0.13's, 300.557'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 34221.294'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.651'MB/sec zip pipeline throughput = 70.651'MB/sec entries = 924598 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.16 avg-zval = 37.37 usrkeys = 924598 avg-key = 20.00 avg-zkey = 1.90 avg-val = 119.30 avg-zval = 37.37 TagRS{ kinds = RS_Key0_TagN bytes = 144560 } TagArray{ size = 282559 bytes = 918336 } seq expand size = 1977913 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5125 value = 3.1351 dict = 1.00 all = 3.4892 } Zip/UnZip{ index = 0.0951 value = 0.3190 dict = 1.00 all = 0.2866 } ---------------------------- total value len = 1.015248 GB avg = 0.120 KB (by entry num) total key len = 0.237511 GB avg = 0.028 KB total ukey len = 0.169651 GB avg = 0.020 KB total ukey num = 0.008482529 Billion total entry num = 0.008482529 Billion write speed all = 0.038599307 MB/sec (with seq num) write speed all = 0.036508435 MB/sec (without seq num) 2024/11/22-17:53:18.609305 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218775: 924598 keys, 38467115 bytes, temperature: kWarm 2024/11/22-17:53:18.609368 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269198609336, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218775, "file_size": 38467115, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3327303738, "table_properties": {"data_size": 34553856, "index_size": 1759040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25888744, "raw_average_key_size": 28, "raw_value_size": 108328984, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924598, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269195, "oldest_key_time": 0, "file_creation_time": 1732269195, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218775, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:53:18.646135 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218776.sst) /dev/shm/ToplingTemp/Topling-228151-jW32HV 2024/11/22-17:53:19.538786 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218776.sst): first pass time = 0.89's, 150.366'MB/sec 2024/11/22-17:53:20.886112 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218776.sst): old prealloc_size = 147639500, real_size = 34246688 2024/11/22-17:53:20.886650 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218776.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925086 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0323 GB avg-val = 117.09 avg-zval = 34.87 2024/11/22-17:53:20.888822 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218776.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:53:21.803593 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218776.sst): second pass time = 1.35's, 80.430'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.789'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3419.827'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 298.004'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33487.456'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.430'MB/sec zip pipeline throughput = 80.430'MB/sec entries = 925086 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.09 avg-zval = 34.87 usrkeys = 925086 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.09 avg-zval = 34.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925086 bytes = 3237808 } seq expand size = 6475602 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5139 value = 3.3580 dict = 1.00 all = 3.5007 } Zip/UnZip{ index = 0.0951 value = 0.2978 dict = 1.00 all = 0.2857 } ---------------------------- total value len = 1.130039 GB avg = 0.120 KB (by entry num) total key len = 0.263413 GB avg = 0.028 KB total ukey len = 0.188152 GB avg = 0.020 KB total ukey num = 0.009407615 Billion total entry num = 0.009407615 Billion write speed all = 0.042929999 MB/sec (with seq num) write speed all = 0.040611332 MB/sec (without seq num) 2024/11/22-17:53:21.839948 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218776: 925086 keys, 38340200 bytes, temperature: kWarm 2024/11/22-17:53:21.840012 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269201839979, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218776, "file_size": 38340200, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701530, "largest_seqno": 3440102052, "table_properties": {"data_size": 32255504, "index_size": 1759744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25902408, "raw_average_key_size": 28, "raw_value_size": 108315360, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925086, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269198, "oldest_key_time": 0, "file_creation_time": 1732269198, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218776, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:53:21.881512 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218777.sst) /dev/shm/ToplingTemp/Topling-228151-8HN9KS 2024/11/22-17:53:22.812605 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218777.sst): first pass time = 0.93's, 144.158'MB/sec 2024/11/22-17:53:24.155660 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218777.sst): old prealloc_size = 147639500, real_size = 31580624 2024/11/22-17:53:24.156216 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218777.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922280 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0296 GB avg-val = 117.53 avg-zval = 32.09 2024/11/22-17:53:24.158368 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218777.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:53:25.022075 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218777.sst): second pass time = 1.34's, 80.749'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 41.167'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3262.915'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.13's, 273.432'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34540.666'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.749'MB/sec zip pipeline throughput = 80.749'MB/sec entries = 922280 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.53 avg-zval = 32.09 usrkeys = 922280 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.53 avg-zval = 32.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922280 bytes = 3228000 } seq expand size = 6455960 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0296 GB dict = 1.08 MB all = 0.0357 GB } UnZip/Zip{ index = 10.5122 value = 3.6626 dict = 1.00 all = 3.7631 } Zip/UnZip{ index = 0.0951 value = 0.2730 dict = 1.00 all = 0.2657 } ---------------------------- total value len = 1.244889 GB avg = 0.121 KB (by entry num) total key len = 0.289237 GB avg = 0.028 KB total ukey len = 0.206598 GB avg = 0.020 KB total ukey num = 0.010329895 Billion total entry num = 0.010329895 Billion write speed all = 0.047259242 MB/sec (with seq num) write speed all = 0.044713516 MB/sec (without seq num) 2024/11/22-17:53:25.061642 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218777: 922280 keys, 35667256 bytes, temperature: kWarm 2024/11/22-17:53:25.061706 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269205061674, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218777, "file_size": 35667256, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701550, "largest_seqno": 3440102712, "table_properties": {"data_size": 29595200, "index_size": 1754688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25823840, "raw_average_key_size": 28, "raw_value_size": 108393936, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922280, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269201, "oldest_key_time": 0, "file_creation_time": 1732269201, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218777, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:53:25.097575 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218778.sst) /dev/shm/ToplingTemp/Topling-228151-CTgmaT 2024/11/22-17:53:26.019470 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218778.sst): first pass time = 0.92's, 145.595'MB/sec 2024/11/22-17:53:27.344621 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218778.sst): old prealloc_size = 147639500, real_size = 32936032 2024/11/22-17:53:27.345179 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218778.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921916 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0310 GB avg-val = 117.59 avg-zval = 33.57 2024/11/22-17:53:27.347384 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218778.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:53:28.237205 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218778.sst): second pass time = 1.32's, 81.842'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 41.478'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3260.742'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.302'MB/sec write SST data time = 0.13's, 287.169'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 32329.872'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.842'MB/sec zip pipeline throughput = 81.842'MB/sec entries = 921916 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.59 avg-zval = 33.57 usrkeys = 921916 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.59 avg-zval = 33.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921916 bytes = 3226720 } seq expand size = 6453412 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5138 value = 3.5024 dict = 1.00 all = 3.6254 } Zip/UnZip{ index = 0.0951 value = 0.2855 dict = 1.00 all = 0.2758 } ---------------------------- total value len = 1.359747 GB avg = 0.121 KB (by entry num) total key len = 0.315051 GB avg = 0.028 KB total ukey len = 0.225036 GB avg = 0.020 KB total ukey num = 0.011251811 Billion total entry num = 0.011251811 Billion write speed all = 0.051587554 MB/sec (with seq num) write speed all = 0.048814904 MB/sec (without seq num) 2024/11/22-17:53:28.271261 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218778: 921916 keys, 37021632 bytes, temperature: kWarm 2024/11/22-17:53:28.271324 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269208271290, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218778, "file_size": 37021632, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3278754630, "largest_seqno": 3440102075, "table_properties": {"data_size": 30951632, "index_size": 1753728, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25813648, "raw_average_key_size": 28, "raw_value_size": 108404148, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921916, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269205, "oldest_key_time": 0, "file_creation_time": 1732269205, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218778, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:53:28.307837 140024777004608 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218779.sst) /dev/shm/ToplingTemp/Topling-228151-Lu8S9U 2024/11/22-17:53:28.539167 140024777004608 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218779.sst): first pass time = 0.23's, 244.052'MB/sec 2024/11/22-17:53:29.086266 140024777004608 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218779.sst): old prealloc_size = 147639500, real_size = 15209552 2024/11/22-17:53:29.086536 140024777004608 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218779.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 392357 prefix = 4 raw-key = 0.0063 GB zip-key = 0.0007 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0455 GB zip-val = 0.0144 GB avg-val = 115.87 avg-zval = 36.61 2024/11/22-17:53:29.086905 140024777004608 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218779.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.056 GB 2024/11/22-17:53:29.466235 140024777004608 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00/218779.sst): second pass time = 0.55's, 83.186'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.38's, 44.226'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3021.003'MB/sec (index lex order gen) rebuild zvType time = 0.32's, 0.302'MB/sec write SST data time = 0.05's, 307.155'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.463'MB, throughput = 13996.433'MB/sec zip my value time = 0.55's, unzip length = 0.045'GB zip my value throughput = 83.186'MB/sec zip pipeline throughput = 83.186'MB/sec entries = 392357 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.87 avg-zval = 36.61 usrkeys = 392357 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.87 avg-zval = 36.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 392357 bytes = 1226128 } seq expand size = 2746499 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0078 GB value = 0.0455 GB dict = 0.46 MB all = 0.0564 GB } __ZipSize{ index = 0.0007 GB value = 0.0144 GB dict = 0.46 MB all = 0.0168 GB } UnZip/Zip{ index = 10.4976 value = 3.1650 dict = 1.00 all = 3.3592 } Zip/UnZip{ index = 0.0953 value = 0.3160 dict = 1.00 all = 0.2977 } ---------------------------- total value len = 1.407954 GB avg = 0.121 KB (by entry num) total key len = 0.326037 GB avg = 0.028 KB total ukey len = 0.232883 GB avg = 0.020 KB total ukey num = 0.011644168 Billion total entry num = 0.011644168 Billion write speed all = 0.053408828 MB/sec (with seq num) write speed all = 0.050539603 MB/sec (without seq num) 2024/11/22-17:53:29.485658 140024777004608 [compaction_job.cc:1948] [default] [JOB 6645] Generated table #218779: 392357 keys, 16803739 bytes, temperature: kWarm 2024/11/22-17:53:29.485721 140024777004608 EVENT_LOG_v1 {"time_micros": 1732269209485688, "cf_name": "default", "job": 6645, "event": "table_file_creation", "file_number": 218779, "file_size": 16803739, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268533, "largest_seqno": 3438582932, "table_properties": {"data_size": 14363776, "index_size": 747520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 10985996, "raw_average_key_size": 28, "raw_value_size": 45461078, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 392357, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732269208, "oldest_key_time": 0, "file_creation_time": 1732269208, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218779, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:53:29.504912 140024777004608 [compaction_job.cc:703] job-06645: subcompact[0], size: 0.468971 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06645/att-00 [218767,218768,218769,218770,218771,218772,218773,218774,218775,218776,218777,218778,218779] 2024/11/22-17:53:29.605473 140024777004608 (Original Log Time 2024/11/22-17:52:40.801045) EVENT_LOG_v1 {"time_micros": 1732269160800985, "job": 6645, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [218762], "files_L3": [218725, 218726, 218727, 218728, 218729, 218730, 218707, 218708, 218709, 218710, 218711, 218712, 218713], "score": 0.946653, "input_data_size": 496694828, "oldest_snapshot_seqno": 3440144425} 2024/11/22-17:53:29.605475 140024777004608 (Original Log Time 2024/11/22-17:53:29.604772) [compaction_job.cc:2022] [default] [JOB 6645] Compacted 1@2 + 13@3 files to L3 => 468971094 bytes 2024/11/22-17:53:29.605477 140024777004608 (Original Log Time 2024/11/22-17:53:29.605406) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 10.2 rd, 9.6 wr, level 3, files in(1, 13) out(13 +0 blob) MB in(27.2, 446.5 +0.0 blob) out(447.2 +0.0 blob), read-write-amplify(33.9) write-amplify(16.5) OK, records in: 11756948, records dropped: 112780 output_compression: Snappy 2024/11/22-17:53:29.605483 140024777004608 (Original Log Time 2024/11/22-17:53:29.605452) EVENT_LOG_v1 {"time_micros": 1732269209605431, "cf": "default", "job": 6645, "event": "compaction_finished", "compaction_time_micros": 48703875, "compaction_time_cpu_micros": 21342971, "output_level": 3, "num_output_files": 13, "total_output_size": 468971094, "num_input_records": 11756948, "num_output_records": 11644168, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 352895431, "file_range_sync_nanos": 0, "file_fsync_nanos": 13356, "file_prepare_write_nanos": 31420, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/22-17:53:29.741911 140024777004608 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732269160,"executesMs":48921,"compactionJobId":6645,"attempt":0,"compactionInputRawBytes":1683075293,"compactionInputZipBytes":496694828,"compactionOutputRawBytes":0,"compactionOutputZipBytes":468971094,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-cda290965a530d853254833db8edc8cc-80d2ef068a885ad2-00"}}, response = 2024/11/22-17:53:29.741982 140024777004608 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-06645/att-00: olev 3, work 48.922 s, result 4.947 ms, install 0.732 ms, input{raw 1.567 GiB zip 473.685 MiB}