2024/11/22-15:23:26.920665 140024701470272 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00: fork to child time = 0.009510 sec 2024/11/22-15:23:26.921452 140024701470272 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00 2024/11/22-15:23:26.928360 140024701470272 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00 2024/11/22-15:23:26.929503 140024701470272 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-15:23:26.929855 140024701470272 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-15:23:26.980246 140024701470272 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-02726/att-00/MANIFEST-206347 2024/11/22-15:23:27.012561 140024701470272 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-02726/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 211051, last_sequence is 3330034610, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 210889 2024/11/22-15:23:27.012568 140024701470272 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-15:23:27.012660 140024701470272 [version_set.cc:5721] Creating manifest 211051 2024/11/22-15:23:27.053931 140024701470272 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-15:23:27.078032 140024701470272 [compaction_job.cc:2437] [default] [JOB 2726] Compacting 1@2 + 13@3 files to L3, score 0.85, subcompactions 1 : 1 2024/11/22-15:23:27.078047 140024701470272 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 2200 Base level 2, inputs: [211042(12MB)], [211047(40MB) 211048(2202KB) 210953(39MB) 210954(39MB) 210955(39MB) 210966(39MB) 210965(39MB) 210967(39MB) 210969(39MB) 210970(39MB) 210968(39MB) 210976(39MB) 210975(33MB)] 2024/11/22-15:23:27.083045 140024701470272 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-15:23:27.083079 140024701470272 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-15:23:27.083097 140024701470272 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-15:23:27.083102 140024701470272 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-15:23:27.083111 140024701470272 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-15:23:27.495410 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211052.sst) /dev/shm/ToplingTemp/Topling-172999-EPuGSy 2024/11/22-15:23:27.656359 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211052.sst): first pass time = 0.16's, 834.146'MB/sec 2024/11/22-15:23:28.664758 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211052.sst): old prealloc_size = 147639500, real_size = 40625664 2024/11/22-15:23:28.664925 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211052.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.72 raw-val = 0.1316 GB zip-val = 0.0404 GB avg-val =1016.99 avg-zval = 311.91 2024/11/22-15:23:28.665051 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211052.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:28.985666 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211052.sst): second pass time = 1.01's, 130.751'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.32's, 131.233'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1893.395'MB/sec (index lex order gen) rebuild zvType time = 0.20's, 0.164'MB/sec write SST data time = 0.12's, 339.600'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 43679.379'MB/sec zip my value time = 1.01's, unzip length = 0.132'GB zip my value throughput = 130.751'MB/sec zip pipeline throughput = 168.657'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.72 avg-val = 1016.99 avg-zval = 311.91 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.72 avg-val = 1018.89 avg-zval = 311.91 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 34980 bytes = 118080 } seq expand size = 244860 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0404 GB dict = 1.38 MB all = 0.0421 GB } UnZip/Zip{ index = 6.9636 value = 3.2606 dict = 1.00 all = 3.1872 } Zip/UnZip{ index = 0.1436 value = 0.3067 dict = 1.00 all = 0.3138 } ---------------------------- total value len = 0.131874 GB avg = 1.019 KB (by entry num) total key len = 0.002589 GB avg = 0.020 KB total ukey len = 0.001553 GB avg = 0.012 KB total ukey num = 0.000129430 Billion total entry num = 0.000129430 Billion write speed all = 0.005730143 MB/sec (with seq num) write speed all = 0.005686018 MB/sec (without seq num) 2024/11/22-15:23:29.024259 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211052: 129430 keys, 42111433 bytes, temperature: kWarm 2024/11/22-15:23:29.024374 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260209024303, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211052, "file_size": 42111433, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329809424, "table_properties": {"data_size": 40370096, "index_size": 223040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629447, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260207, "oldest_key_time": 0, "file_creation_time": 1732260207, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211052, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:29.073829 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211053.sst) /dev/shm/ToplingTemp/Topling-172999-OoRMkA 2024/11/22-15:23:29.653823 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211053.sst): first pass time = 0.58's, 231.431'MB/sec 2024/11/22-15:23:30.445443 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211053.sst): old prealloc_size = 147639500, real_size = 40367984 2024/11/22-15:23:30.445581 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211053.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129429 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.71 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.00 avg-zval = 309.93 2024/11/22-15:23:30.445666 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211053.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:30.783629 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211053.sst): second pass time = 0.79's, 166.424'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.34's, 123.447'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1797.370'MB/sec (index lex order gen) rebuild zvType time = 0.26's, 0.123'MB/sec write SST data time = 0.07's, 556.658'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 43441.520'MB/sec zip my value time = 0.79's, unzip length = 0.132'GB zip my value throughput = 166.424'MB/sec zip pipeline throughput = 166.424'MB/sec entries = 129429 avg-key = 20.00 avg-zkey = 1.71 avg-val = 1017.00 avg-zval = 309.93 usrkeys = 129429 avg-key = 12.00 avg-zkey = 1.71 avg-val = 1017.14 avg-zval = 309.93 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 2612 bytes = 8496 } seq expand size = 18284 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0281 value = 3.2813 dict = 1.00 all = 3.2153 } Zip/UnZip{ index = 0.1423 value = 0.3048 dict = 1.00 all = 0.3110 } ---------------------------- total value len = 0.263522 GB avg = 1.018 KB (by entry num) total key len = 0.005177 GB avg = 0.020 KB total ukey len = 0.003106 GB avg = 0.012 KB total ukey num = 0.000258859 Billion total entry num = 0.000258859 Billion write speed all = 0.011449756 MB/sec (with seq num) write speed all = 0.011361512 MB/sec (without seq num) 2024/11/22-15:23:30.824068 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211053: 129429 keys, 41744172 bytes, temperature: kWarm 2024/11/22-15:23:30.824147 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260210824106, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211053, "file_size": 41744172, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329800648, "table_properties": {"data_size": 40114464, "index_size": 220992, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588580, "raw_average_key_size": 20, "raw_value_size": 131629519, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129429, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260209, "oldest_key_time": 0, "file_creation_time": 1732260209, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211053, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:30.910612 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211054.sst) /dev/shm/ToplingTemp/Topling-172999-EWvyny 2024/11/22-15:23:31.400804 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211054.sst): first pass time = 0.49's, 273.836'MB/sec 2024/11/22-15:23:32.240414 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211054.sst): old prealloc_size = 147639500, real_size = 40483312 2024/11/22-15:23:32.240551 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211054.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129425 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.70 raw-val = 0.1316 GB zip-val = 0.0402 GB avg-val =1017.03 avg-zval = 310.85 2024/11/22-15:23:32.240623 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211054.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:32.512794 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211054.sst): second pass time = 0.84's, 156.897'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.27's, 153.690'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1951.282'MB/sec (index lex order gen) rebuild zvType time = 0.21's, 0.156'MB/sec write SST data time = 0.06's, 651.545'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 44696.569'MB/sec zip my value time = 0.84's, unzip length = 0.132'GB zip my value throughput = 156.897'MB/sec zip pipeline throughput = 156.897'MB/sec entries = 129425 avg-key = 20.00 avg-zkey = 1.70 avg-val = 1017.03 avg-zval = 310.85 usrkeys = 129425 avg-key = 12.00 avg-zkey = 1.70 avg-val = 1017.08 avg-zval = 310.85 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 854 bytes = 2688 } seq expand size = 5978 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0402 GB dict = 1.38 MB all = 0.0419 GB } UnZip/Zip{ index = 7.0709 value = 3.2718 dict = 1.00 all = 3.2068 } Zip/UnZip{ index = 0.1414 value = 0.3056 dict = 1.00 all = 0.3118 } ---------------------------- total value len = 0.395158 GB avg = 1.018 KB (by entry num) total key len = 0.007766 GB avg = 0.020 KB total ukey len = 0.004659 GB avg = 0.012 KB total ukey num = 0.000388284 Billion total entry num = 0.000388284 Billion write speed all = 0.017168017 MB/sec (with seq num) write speed all = 0.017035663 MB/sec (without seq num) 2024/11/22-15:23:32.553548 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211054: 129425 keys, 41853885 bytes, temperature: kWarm 2024/11/22-15:23:32.553767 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260212553584, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211054, "file_size": 41853885, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329775260, "table_properties": {"data_size": 40231136, "index_size": 219648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588500, "raw_average_key_size": 20, "raw_value_size": 131629569, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129425, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260210, "oldest_key_time": 0, "file_creation_time": 1732260210, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211054, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:32.632640 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211055.sst) /dev/shm/ToplingTemp/Topling-172999-Egp47y 2024/11/22-15:23:33.167019 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211055.sst): first pass time = 0.53's, 251.189'MB/sec 2024/11/22-15:23:33.959776 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211055.sst): old prealloc_size = 147639500, real_size = 40355184 2024/11/22-15:23:33.959896 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211055.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129431 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.99 avg-zval = 309.85 2024/11/22-15:23:33.959945 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211055.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:34.113174 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211055.sst): second pass time = 0.79's, 166.291'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 272.006'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2409.794'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.308'MB/sec write SST data time = 0.05's, 867.364'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 44144.821'MB/sec zip my value time = 0.79's, unzip length = 0.132'GB zip my value throughput = 166.291'MB/sec zip pipeline throughput = 166.291'MB/sec entries = 129431 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 usrkeys = 129431 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 136 bytes = 416 } seq expand size = 952 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0898 value = 3.2822 dict = 1.00 all = 3.2169 } Zip/UnZip{ index = 0.1410 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.526788 GB avg = 1.018 KB (by entry num) total key len = 0.010354 GB avg = 0.020 KB total ukey len = 0.006213 GB avg = 0.012 KB total ukey num = 0.000517715 Billion total entry num = 0.000517715 Billion write speed all = 0.022885353 MB/sec (with seq num) write speed all = 0.022708892 MB/sec (without seq num) 2024/11/22-15:23:34.150968 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211055: 129431 keys, 41723147 bytes, temperature: kWarm 2024/11/22-15:23:34.151017 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260214150990, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211055, "file_size": 41723147, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329803501, "table_properties": {"data_size": 40103584, "index_size": 219072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588620, "raw_average_key_size": 20, "raw_value_size": 131629614, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260212, "oldest_key_time": 0, "file_creation_time": 1732260212, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211055, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:34.188146 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211056.sst) /dev/shm/ToplingTemp/Topling-172999-rraW7A 2024/11/22-15:23:34.645358 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211056.sst): first pass time = 0.46's, 293.581'MB/sec 2024/11/22-15:23:36.014200 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211056.sst): old prealloc_size = 147639500, real_size = 40354832 2024/11/22-15:23:36.014311 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211056.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.99 avg-zval = 309.85 2024/11/22-15:23:36.014366 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211056.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:36.166298 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211056.sst): second pass time = 1.37's, 96.197'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 274.328'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2448.044'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.310'MB/sec write SST data time = 0.05's, 877.967'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 46811.557'MB/sec zip my value time = 1.37's, unzip length = 0.132'GB zip my value throughput = 96.197'MB/sec zip pipeline throughput = 137.559'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 10 bytes = 48 } seq expand size = 70 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2822 dict = 1.00 all = 3.2169 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.658418 GB avg = 1.017 KB (by entry num) total key len = 0.012943 GB avg = 0.020 KB total ukey len = 0.007766 GB avg = 0.012 KB total ukey num = 0.000647145 Billion total entry num = 0.000647145 Billion write speed all = 0.028601310 MB/sec (with seq num) write speed all = 0.028380752 MB/sec (without seq num) 2024/11/22-15:23:36.202742 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211056: 129430 keys, 41722520 bytes, temperature: kWarm 2024/11/22-15:23:36.202806 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260216202766, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211056, "file_size": 41722520, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329386734, "table_properties": {"data_size": 40103488, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629362, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260214, "oldest_key_time": 0, "file_creation_time": 1732260214, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211056, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:36.239110 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211057.sst) /dev/shm/ToplingTemp/Topling-172999-CTqGAz 2024/11/22-15:23:36.787228 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211057.sst): first pass time = 0.55's, 244.885'MB/sec 2024/11/22-15:23:37.597460 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211057.sst): old prealloc_size = 147639500, real_size = 40358608 2024/11/22-15:23:37.597574 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211057.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129428 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.01 avg-zval = 309.88 2024/11/22-15:23:37.597624 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211057.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:37.749631 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211057.sst): second pass time = 0.81's, 162.775'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 274.230'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2393.891'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.312'MB/sec write SST data time = 0.05's, 864.087'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 42682.310'MB/sec zip my value time = 0.81's, unzip length = 0.132'GB zip my value throughput = 162.775'MB/sec zip pipeline throughput = 162.775'MB/sec entries = 129428 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.01 avg-zval = 309.88 usrkeys = 129428 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.01 avg-zval = 309.88 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 13 bytes = 48 } seq expand size = 91 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0979 value = 3.2819 dict = 1.00 all = 3.2166 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.790047 GB avg = 1.017 KB (by entry num) total key len = 0.015531 GB avg = 0.020 KB total ukey len = 0.009319 GB avg = 0.012 KB total ukey num = 0.000776573 Billion total entry num = 0.000776573 Billion write speed all = 0.034316952 MB/sec (with seq num) write speed all = 0.034052301 MB/sec (without seq num) 2024/11/22-15:23:37.786281 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211057: 129428 keys, 41726264 bytes, temperature: kWarm 2024/11/22-15:23:37.786340 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260217786309, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211057, "file_size": 41726264, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329796401, "table_properties": {"data_size": 40107264, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588560, "raw_average_key_size": 20, "raw_value_size": 131629347, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129428, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260216, "oldest_key_time": 0, "file_creation_time": 1732260216, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211057, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:37.822298 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211058.sst) /dev/shm/ToplingTemp/Topling-172999-dqS76z 2024/11/22-15:23:38.394278 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211058.sst): first pass time = 0.57's, 234.668'MB/sec 2024/11/22-15:23:39.178380 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211058.sst): old prealloc_size = 147639500, real_size = 40354848 2024/11/22-15:23:39.178498 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211058.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129430 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.99 avg-zval = 309.85 2024/11/22-15:23:39.178550 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211058.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:39.332472 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211058.sst): second pass time = 0.78's, 168.278'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 270.804'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2365.783'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.310'MB/sec write SST data time = 0.05's, 838.720'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 44614.901'MB/sec zip my value time = 0.78's, unzip length = 0.132'GB zip my value throughput = 168.278'MB/sec zip pipeline throughput = 168.278'MB/sec entries = 129430 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 usrkeys = 129430 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.85 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 15 bytes = 48 } seq expand size = 105 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0980 value = 3.2822 dict = 1.00 all = 3.2169 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 0.921677 GB avg = 1.017 KB (by entry num) total key len = 0.018120 GB avg = 0.020 KB total ukey len = 0.010872 GB avg = 0.012 KB total ukey num = 0.000906003 Billion total entry num = 0.000906003 Billion write speed all = 0.040031828 MB/sec (with seq num) write speed all = 0.039723089 MB/sec (without seq num) 2024/11/22-15:23:39.368762 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211058: 129430 keys, 41722417 bytes, temperature: kWarm 2024/11/22-15:23:39.368830 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260219368799, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211058, "file_size": 41722417, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329436082, "table_properties": {"data_size": 40103504, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588600, "raw_average_key_size": 20, "raw_value_size": 131629403, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129430, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260217, "oldest_key_time": 0, "file_creation_time": 1732260217, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211058, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:39.408043 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211059.sst) /dev/shm/ToplingTemp/Topling-172999-HZ0FTB 2024/11/22-15:23:39.888317 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211059.sst): first pass time = 0.48's, 279.484'MB/sec 2024/11/22-15:23:40.824462 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211059.sst): old prealloc_size = 147639500, real_size = 40360192 2024/11/22-15:23:40.824592 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211059.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129426 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.03 avg-zval = 309.90 2024/11/22-15:23:40.824677 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211059.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:41.023562 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211059.sst): second pass time = 0.94's, 140.704'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.20's, 209.593'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1775.182'MB/sec (index lex order gen) rebuild zvType time = 0.15's, 0.220'MB/sec write SST data time = 0.05's, 808.775'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 33857.691'MB/sec zip my value time = 0.94's, unzip length = 0.132'GB zip my value throughput = 140.704'MB/sec zip pipeline throughput = 192.199'MB/sec entries = 129426 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.90 usrkeys = 129426 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.90 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 15 bytes = 48 } seq expand size = 105 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0978 value = 3.2818 dict = 1.00 all = 3.2165 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 1.053307 GB avg = 1.017 KB (by entry num) total key len = 0.020709 GB avg = 0.020 KB total ukey len = 0.012425 GB avg = 0.012 KB total ukey num = 0.001035429 Billion total entry num = 0.001035429 Billion write speed all = 0.045745752 MB/sec (with seq num) write speed all = 0.045392934 MB/sec (without seq num) 2024/11/22-15:23:41.059125 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211059: 129426 keys, 41727722 bytes, temperature: kWarm 2024/11/22-15:23:41.059183 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260221059154, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211059, "file_size": 41727722, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329796375, "table_properties": {"data_size": 40108848, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588520, "raw_average_key_size": 20, "raw_value_size": 131630155, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129426, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260219, "oldest_key_time": 0, "file_creation_time": 1732260219, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211059, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:41.097025 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211060.sst) /dev/shm/ToplingTemp/Topling-172999-ZGOJjB 2024/11/22-15:23:41.577252 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211060.sst): first pass time = 0.48's, 279.508'MB/sec 2024/11/22-15:23:42.358641 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211060.sst): old prealloc_size = 147639500, real_size = 40354784 2024/11/22-15:23:42.358756 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211060.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129431 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.99 avg-zval = 309.84 2024/11/22-15:23:42.358819 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211060.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:42.513481 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211060.sst): second pass time = 0.78's, 168.705'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 269.474'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2385.358'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.304'MB/sec write SST data time = 0.05's, 865.802'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 43661.270'MB/sec zip my value time = 0.78's, unzip length = 0.132'GB zip my value throughput = 168.705'MB/sec zip pipeline throughput = 168.705'MB/sec entries = 129431 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.84 usrkeys = 129431 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.99 avg-zval = 309.84 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 11 bytes = 48 } seq expand size = 77 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0981 value = 3.2822 dict = 1.00 all = 3.2169 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 1.184936 GB avg = 1.017 KB (by entry num) total key len = 0.023297 GB avg = 0.020 KB total ukey len = 0.013978 GB avg = 0.012 KB total ukey num = 0.001164860 Billion total entry num = 0.001164860 Billion write speed all = 0.051459268 MB/sec (with seq num) write speed all = 0.051062372 MB/sec (without seq num) 2024/11/22-15:23:42.549471 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211060: 129431 keys, 41722548 bytes, temperature: kWarm 2024/11/22-15:23:42.549526 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260222549498, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211060, "file_size": 41722548, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329799864, "table_properties": {"data_size": 40103440, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588620, "raw_average_key_size": 20, "raw_value_size": 131629488, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260221, "oldest_key_time": 0, "file_creation_time": 1732260221, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211060, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:42.585992 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211061.sst) /dev/shm/ToplingTemp/Topling-172999-KaLJaA 2024/11/22-15:23:43.035659 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211061.sst): first pass time = 0.45's, 298.507'MB/sec 2024/11/22-15:23:43.858856 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211061.sst): old prealloc_size = 147639500, real_size = 40351168 2024/11/22-15:23:43.858979 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211061.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129436 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1016.94 avg-zval = 309.80 2024/11/22-15:23:43.859040 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211061.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:44.011018 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211061.sst): second pass time = 0.82's, 160.037'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 274.195'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2419.674'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.313'MB/sec write SST data time = 0.05's, 858.467'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 43152.897'MB/sec zip my value time = 0.82's, unzip length = 0.132'GB zip my value throughput = 160.037'MB/sec zip pipeline throughput = 160.037'MB/sec entries = 129436 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1016.94 avg-zval = 309.80 usrkeys = 129436 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1016.95 avg-zval = 309.80 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 16 bytes = 64 } seq expand size = 112 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0983 value = 3.2825 dict = 1.00 all = 3.2172 } Zip/UnZip{ index = 0.1409 value = 0.3046 dict = 1.00 all = 0.3108 } ---------------------------- total value len = 1.316566 GB avg = 1.017 KB (by entry num) total key len = 0.025886 GB avg = 0.020 KB total ukey len = 0.015532 GB avg = 0.012 KB total ukey num = 0.001294296 Billion total entry num = 0.001294296 Billion write speed all = 0.057172037 MB/sec (with seq num) write speed all = 0.056731068 MB/sec (without seq num) 2024/11/22-15:23:44.049362 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211061: 129436 keys, 41719360 bytes, temperature: kWarm 2024/11/22-15:23:44.049426 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260224049393, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211061, "file_size": 41719360, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329746124, "table_properties": {"data_size": 40099824, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588720, "raw_average_key_size": 20, "raw_value_size": 131629284, "raw_average_value_size": 1016, "num_data_blocks": 1, "num_entries": 129436, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260222, "oldest_key_time": 0, "file_creation_time": 1732260222, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211061, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:44.091554 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211062.sst) /dev/shm/ToplingTemp/Topling-172999-2Be0rz 2024/11/22-15:23:44.624739 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211062.sst): first pass time = 0.53's, 251.749'MB/sec 2024/11/22-15:23:45.420015 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211062.sst): old prealloc_size = 147639500, real_size = 40358608 2024/11/22-15:23:45.420132 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211062.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 129426 prefix = 4 raw-key = 0.0010 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1316 GB zip-val = 0.0401 GB avg-val =1017.03 avg-zval = 309.89 2024/11/22-15:23:45.420184 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211062.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-15:23:45.568581 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211062.sst): second pass time = 0.79's, 165.640'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 280.923'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2345.974'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.315'MB/sec write SST data time = 0.05's, 911.747'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.378'MB, throughput = 38109.311'MB/sec zip my value time = 0.79's, unzip length = 0.132'GB zip my value throughput = 165.640'MB/sec zip pipeline throughput = 165.640'MB/sec entries = 129426 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.89 usrkeys = 129426 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.03 avg-zval = 309.89 TagRS{ kinds = RS_Key0_TagN bytes = 20320 } TagArray{ size = 13 bytes = 48 } seq expand size = 91 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0016 GB value = 0.1316 GB dict = 1.38 MB all = 0.1342 GB } __ZipSize{ index = 0.0002 GB value = 0.0401 GB dict = 1.38 MB all = 0.0417 GB } UnZip/Zip{ index = 7.0978 value = 3.2819 dict = 1.00 all = 3.2166 } Zip/UnZip{ index = 0.1409 value = 0.3047 dict = 1.00 all = 0.3109 } ---------------------------- total value len = 1.448195 GB avg = 1.017 KB (by entry num) total key len = 0.028474 GB avg = 0.020 KB total ukey len = 0.017085 GB avg = 0.012 KB total ukey num = 0.001423722 Billion total entry num = 0.001423722 Billion write speed all = 0.062883919 MB/sec (with seq num) write speed all = 0.062398886 MB/sec (without seq num) 2024/11/22-15:23:45.607164 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211062: 129426 keys, 41726498 bytes, temperature: kWarm 2024/11/22-15:23:45.607224 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260225607192, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211062, "file_size": 41726498, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329714457, "table_properties": {"data_size": 40107264, "index_size": 218816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2588520, "raw_average_key_size": 20, "raw_value_size": 131629552, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 129426, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260224, "oldest_key_time": 0, "file_creation_time": 1732260224, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211062, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:45.641299 140024701470272 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211063.sst) /dev/shm/ToplingTemp/Topling-172999-0EUhOz 2024/11/22-15:23:46.140499 140024701470272 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211063.sst): first pass time = 0.50's, 240.811'MB/sec 2024/11/22-15:23:46.866714 140024701470272 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211063.sst): old prealloc_size = 147639500, real_size = 36164976 2024/11/22-15:23:46.866834 140024701470272 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211063.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 115913 prefix = 4 raw-key = 0.0009 GB zip-key = 0.0002 GB avg-key = 8.00 avg-zkey = 1.69 raw-val = 0.1179 GB zip-val = 0.0359 GB avg-val =1017.04 avg-zval = 310.06 2024/11/22-15:23:46.866884 140024701470272 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211063.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.000 sec, Key+Value = 0.120 GB 2024/11/22-15:23:47.002832 140024701470272 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00/211063.sst): second pass time = 0.73's, 162.602'MB/sec, value only(98.1% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.14's, 274.716'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2060.786'MB/sec (index lex order gen) rebuild zvType time = 0.09's, 0.312'MB/sec write SST data time = 0.04's, 869.845'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.231'MB, throughput = 36535.687'MB/sec zip my value time = 0.73's, unzip length = 0.118'GB zip my value throughput = 162.602'MB/sec zip pipeline throughput = 162.602'MB/sec entries = 115913 avg-key = 20.00 avg-zkey = 1.69 avg-val = 1017.04 avg-zval = 310.06 usrkeys = 115913 avg-key = 12.00 avg-zkey = 1.69 avg-val = 1017.04 avg-zval = 310.06 TagRS{ kinds = RS_Key0_TagN bytes = 18200 } TagArray{ size = 9 bytes = 32 } seq expand size = 63 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0014 GB value = 0.1179 GB dict = 1.23 MB all = 0.1202 GB } __ZipSize{ index = 0.0002 GB value = 0.0359 GB dict = 1.23 MB all = 0.0374 GB } UnZip/Zip{ index = 7.0909 value = 3.2802 dict = 1.00 all = 3.2152 } Zip/UnZip{ index = 0.1410 value = 0.3049 dict = 1.00 all = 0.3110 } ---------------------------- total value len = 1.566083 GB avg = 1.017 KB (by entry num) total key len = 0.030793 GB avg = 0.020 KB total ukey len = 0.018476 GB avg = 0.012 KB total ukey num = 0.001539635 Billion total entry num = 0.001539635 Billion write speed all = 0.067998730 MB/sec (with seq num) write speed all = 0.067474240 MB/sec (without seq num) 2024/11/22-15:23:47.036585 140024701470272 [compaction_job.cc:1948] [default] [JOB 2726] Generated table #211063: 115913 keys, 37386562 bytes, temperature: kWarm 2024/11/22-15:23:47.036645 140024701470272 EVENT_LOG_v1 {"time_micros": 1732260227036615, "cf_name": "default", "job": 2726, "event": "table_file_creation", "file_number": 211063, "file_size": 37386562, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3329547230, "table_properties": {"data_size": 35939680, "index_size": 196160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 2318260, "raw_average_key_size": 20, "raw_value_size": 117887616, "raw_average_value_size": 1017, "num_data_blocks": 1, "num_entries": 115913, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 20, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732260225, "oldest_key_time": 0, "file_creation_time": 1732260225, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 211063, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-15:23:47.060602 140024701470272 [compaction_job.cc:703] job-02726: subcompact[0], size: 0.496887 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-02726/att-00 [211052,211053,211054,211055,211056,211057,211058,211059,211060,211061,211062,211063] 2024/11/22-15:23:47.141257 140024701470272 (Original Log Time 2024/11/22-15:23:27.078091) EVENT_LOG_v1 {"time_micros": 1732260207078057, "job": 2726, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [211042], "files_L3": [211047, 211048, 210953, 210954, 210955, 210966, 210965, 210967, 210969, 210970, 210968, 210976, 210975], "score": 0.853124, "input_data_size": 509703919, "oldest_snapshot_seqno": 3330034568} 2024/11/22-15:23:47.141260 140024701470272 (Original Log Time 2024/11/22-15:23:47.140587) [compaction_job.cc:2022] [default] [JOB 2726] Compacted 1@2 + 13@3 files to L3 => 496886528 bytes 2024/11/22-15:23:47.141262 140024701470272 (Original Log Time 2024/11/22-15:23:47.141196) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 25.5 rd, 24.9 wr, level 3, files in(1, 13) out(12 +0 blob) MB in(12.4, 473.7 +0.0 blob) out(473.9 +0.0 blob), read-write-amplify(77.2) write-amplify(38.1) OK, records in: 1551873, records dropped: 12238 output_compression: Snappy 2024/11/22-15:23:47.141268 140024701470272 (Original Log Time 2024/11/22-15:23:47.141239) EVENT_LOG_v1 {"time_micros": 1732260227141219, "cf": "default", "job": 2726, "event": "compaction_finished", "compaction_time_micros": 19982520, "compaction_time_cpu_micros": 4226343, "output_level": 3, "num_output_files": 12, "total_output_size": 496886528, "num_input_records": 1551873, "num_output_records": 1539635, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 421076909, "file_range_sync_nanos": 0, "file_fsync_nanos": 10504, "file_prepare_write_nanos": 23371, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/22-15:23:47.282936 140024701470272 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732260206,"executesMs":20214,"compactionJobId":2726,"attempt":0,"compactionInputRawBytes":1609296579,"compactionInputZipBytes":509703919,"compactionOutputRawBytes":0,"compactionOutputZipBytes":496886528,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-be77a5c101ab9f37f557320024b9c19c-ee74a096dc3cc5e7-00"}}, response = 2024/11/22-15:23:47.283009 140024701470272 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-02726/att-00: olev 3, work 20.215 s, result 4.952 ms, install 0.710 ms, input{raw 1.499 GiB zip 486.092 MiB}