2024/11/23-06:17:45.708388 140025196639808 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00: fork to child time = 0.010066 sec 2024/11/23-06:17:45.709184 140025196639808 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00 2024/11/23-06:17:45.716442 140025196639808 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00 2024/11/23-06:17:45.717578 140025196639808 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-06:17:45.717854 140025196639808 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-06:17:45.765611 140025196639808 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-20880/att-00/MANIFEST-206347 2024/11/23-06:17:45.782807 140025196639808 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-20880/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 248213, last_sequence is 3831438970, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 248085 2024/11/23-06:17:45.782816 140025196639808 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-06:17:45.783030 140025196639808 [version_set.cc:5721] Creating manifest 248213 2024/11/23-06:17:45.814017 140025196639808 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-06:17:45.832712 140025196639808 [compaction_job.cc:2437] [default] [JOB 20880] Compacting 1@2 + 13@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-06:17:45.832732 140025196639808 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 15623 Base level 2, inputs: [248209(27MB)], [248172(35MB) 248173(35MB) 248174(37MB) 248175(34MB) 248176(7959KB) 248154(36MB) 248155(36MB) 248156(36MB) 248157(35MB) 248158(35MB) 248159(35MB) 248160(35MB) 248161(35MB)] 2024/11/23-06:17:45.840141 140025196639808 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-06:17:45.840177 140025196639808 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-06:17:45.840198 140025196639808 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-06:17:45.840203 140025196639808 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-06:17:45.840217 140025196639808 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-06:17:46.359639 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248214.sst) /dev/shm/ToplingTemp/Topling-432069-7rhoLc 2024/11/23-06:17:47.350439 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248214.sst): first pass time = 0.99's, 135.470'MB/sec 2024/11/23-06:17:49.489183 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248214.sst): old prealloc_size = 147639500, real_size = 33092128 2024/11/23-06:17:49.489760 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248214.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929555 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.39 avg-zval = 33.45 2024/11/23-06:17:49.490561 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248214.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:17:50.959384 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248214.sst): second pass time = 2.14's, 50.604'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.47's, 25.302'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3166.981'MB/sec (index lex order gen) rebuild zvType time = 1.02's, 0.229'MB/sec write SST data time = 0.45's, 81.931'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33977.783'MB/sec zip my value time = 2.14's, unzip length = 0.108'GB zip my value throughput = 50.604'MB/sec zip pipeline throughput = 112.172'MB/sec entries = 929555 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.39 avg-zval = 33.45 usrkeys = 929555 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.39 avg-zval = 33.45 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929555 bytes = 3253456 } seq expand size = 6506885 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5123 value = 3.4798 dict = 1.00 all = 3.6081 } Zip/UnZip{ index = 0.0951 value = 0.2874 dict = 1.00 all = 0.2772 } ---------------------------- total value len = 0.114697 GB avg = 0.123 KB (by entry num) total key len = 0.026028 GB avg = 0.028 KB total ukey len = 0.018591 GB avg = 0.020 KB total ukey num = 0.000929555 Billion total entry num = 0.000929555 Billion write speed all = 0.001824563 MB/sec (with seq num) write speed all = 0.001728146 MB/sec (without seq num) 2024/11/23-06:17:51.006849 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248214: 929555 keys, 37198704 bytes, temperature: kWarm 2024/11/23-06:17:51.006967 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313871006893, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248214, "file_size": 37198704, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3621075956, "largest_seqno": 3831419147, "table_properties": {"data_size": 31091024, "index_size": 1768512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26027540, "raw_average_key_size": 28, "raw_value_size": 108190246, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929555, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313866, "oldest_key_time": 0, "file_creation_time": 1732313866, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248214, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:17:51.073207 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248215.sst) /dev/shm/ToplingTemp/Topling-432069-V4W8Ee 2024/11/23-06:17:52.272660 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248215.sst): first pass time = 1.20's, 111.905'MB/sec 2024/11/23-06:17:53.952536 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248215.sst): old prealloc_size = 147639500, real_size = 33088656 2024/11/23-06:17:53.953138 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248215.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927669 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.68 avg-zval = 33.52 2024/11/23-06:17:53.953960 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248215.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:17:55.108705 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248215.sst): second pass time = 1.68's, 64.468'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.16's, 32.267'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3041.038'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.254'MB/sec write SST data time = 0.24's, 153.937'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 11143.438'MB/sec zip my value time = 1.68's, unzip length = 0.108'GB zip my value throughput = 64.468'MB/sec zip pipeline throughput = 64.468'MB/sec entries = 927669 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.68 avg-zval = 33.52 usrkeys = 927669 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.68 avg-zval = 33.52 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927669 bytes = 3362816 } seq expand size = 6493683 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5138 value = 3.4814 dict = 1.00 all = 3.5978 } Zip/UnZip{ index = 0.0951 value = 0.2872 dict = 1.00 all = 0.2779 } ---------------------------- total value len = 0.229434 GB avg = 0.124 KB (by entry num) total key len = 0.052002 GB avg = 0.028 KB total ukey len = 0.037144 GB avg = 0.020 KB total ukey num = 0.001857224 Billion total entry num = 0.001857224 Billion write speed all = 0.003648759 MB/sec (with seq num) write speed all = 0.003456131 MB/sec (without seq num) 2024/11/23-06:17:55.143827 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248215: 927669 keys, 37305290 bytes, temperature: kWarm 2024/11/23-06:17:55.143909 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313875143866, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248215, "file_size": 37305290, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3831419216, "table_properties": {"data_size": 31091904, "index_size": 1764672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25974732, "raw_average_key_size": 28, "raw_value_size": 108243030, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927669, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313871, "oldest_key_time": 0, "file_creation_time": 1732313871, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248215, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:17:55.210048 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248216.sst) /dev/shm/ToplingTemp/Topling-432069-UGuX2d 2024/11/23-06:17:56.295459 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248216.sst): first pass time = 1.09's, 123.662'MB/sec 2024/11/23-06:17:57.670722 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248216.sst): old prealloc_size = 147639500, real_size = 34992176 2024/11/23-06:17:57.671274 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248216.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931500 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0330 GB avg-val = 116.09 avg-zval = 35.41 2024/11/23-06:17:57.672044 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248216.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:17:58.564390 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248216.sst): second pass time = 1.37's, 78.669'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 43.757'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3350.435'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.304'MB/sec write SST data time = 0.13's, 306.429'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 35941.245'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.669'MB/sec zip pipeline throughput = 78.669'MB/sec entries = 931500 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.09 avg-zval = 35.41 usrkeys = 931500 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.09 avg-zval = 35.41 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931500 bytes = 3260272 } seq expand size = 6520500 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.5137 value = 3.2781 dict = 1.00 all = 3.4324 } Zip/UnZip{ index = 0.0951 value = 0.3051 dict = 1.00 all = 0.2913 } ---------------------------- total value len = 0.344090 GB avg = 0.123 KB (by entry num) total key len = 0.078084 GB avg = 0.028 KB total ukey len = 0.055774 GB avg = 0.020 KB total ukey num = 0.002788724 Billion total entry num = 0.002788724 Billion write speed all = 0.005473156 MB/sec (with seq num) write speed all = 0.005183928 MB/sec (without seq num) 2024/11/23-06:17:58.604811 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248216: 931500 keys, 39103266 bytes, temperature: kWarm 2024/11/23-06:17:58.604876 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313878604843, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248216, "file_size": 39103266, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3563652023, "largest_seqno": 3831418555, "table_properties": {"data_size": 32987168, "index_size": 1771968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26082000, "raw_average_key_size": 28, "raw_value_size": 108135852, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931500, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313875, "oldest_key_time": 0, "file_creation_time": 1732313875, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248216, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:17:58.647856 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248217.sst) /dev/shm/ToplingTemp/Topling-432069-JIvHPe 2024/11/23-06:17:59.728178 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248217.sst): first pass time = 1.08's, 124.243'MB/sec 2024/11/23-06:18:01.235183 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248217.sst): old prealloc_size = 147639500, real_size = 33625408 2024/11/23-06:18:01.235986 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248217.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1088189 prefix = 4 raw-key = 0.0174 GB zip-key = 0.0021 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1037 GB zip-val = 0.0313 GB avg-val = 95.34 avg-zval = 28.72 2024/11/23-06:18:01.236427 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248217.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0029 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:18:01.978045 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248217.sst): second pass time = 1.51's, 68.881'MB/sec, value only(77.3% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.74's, 48.090'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2683.861'MB/sec (index lex order gen) rebuild zvType time = 0.60's, 0.454'MB/sec write SST data time = 0.14's, 251.056'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.037'MB, throughput = 31464.963'MB/sec zip my value time = 1.51's, unzip length = 0.104'GB zip my value throughput = 68.881'MB/sec zip pipeline throughput = 68.881'MB/sec entries = 1088189 avg-key = 28.00 avg-zkey = 1.93 avg-val = 95.34 avg-zval = 28.72 usrkeys = 1088189 avg-key = 20.00 avg-zkey = 1.93 avg-val = 97.62 avg-zval = 28.72 TagRS{ kinds = RS_Key0_TagN bytes = 170120 } TagArray{ size = 354533 bytes = 1152240 } seq expand size = 2481731 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0218 GB value = 0.1037 GB dict = 1.04 MB all = 0.1342 GB } __ZipSize{ index = 0.0021 GB value = 0.0313 GB dict = 1.04 MB all = 0.0357 GB } UnZip/Zip{ index = 10.3787 value = 3.3193 dict = 1.00 all = 3.7571 } Zip/UnZip{ index = 0.0964 value = 0.3013 dict = 1.00 all = 0.2662 } ---------------------------- total value len = 0.450320 GB avg = 0.116 KB (by entry num) total key len = 0.108554 GB avg = 0.028 KB total ukey len = 0.077538 GB avg = 0.020 KB total ukey num = 0.003876913 Billion total entry num = 0.003876913 Billion write speed all = 0.007245037 MB/sec (with seq num) write speed all = 0.006842966 MB/sec (without seq num) 2024/11/23-06:18:02.010900 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248217: 1088189 keys, 35723601 bytes, temperature: kWarm 2024/11/23-06:18:02.010965 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313882010930, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248217, "file_size": 35723601, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3831418664, "table_properties": {"data_size": 31256240, "index_size": 2096960, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 30469292, "raw_average_key_size": 28, "raw_value_size": 103748543, "raw_average_value_size": 95, "num_data_blocks": 1, "num_entries": 1088189, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313878, "oldest_key_time": 0, "file_creation_time": 1732313878, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248217, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:18:02.046889 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248218.sst) /dev/shm/ToplingTemp/Topling-432069-09Tx7d 2024/11/23-06:18:03.078501 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248218.sst): first pass time = 1.03's, 130.110'MB/sec 2024/11/23-06:18:04.931090 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248218.sst): old prealloc_size = 147639500, real_size = 36623616 2024/11/23-06:18:04.931651 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248218.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924651 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 117.16 avg-zval = 37.46 2024/11/23-06:18:04.931994 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248218.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:18:05.806106 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248218.sst): second pass time = 1.85's, 58.493'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 44.084'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3254.207'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.363'MB/sec write SST data time = 0.24's, 161.811'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33240.523'MB/sec zip my value time = 1.85's, unzip length = 0.108'GB zip my value throughput = 58.493'MB/sec zip pipeline throughput = 111.644'MB/sec entries = 924651 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.16 avg-zval = 37.46 usrkeys = 924651 avg-key = 20.00 avg-zkey = 1.90 avg-val = 119.20 avg-zval = 37.46 TagRS{ kinds = RS_Key0_TagN bytes = 144560 } TagArray{ size = 269904 bytes = 944672 } seq expand size = 1889328 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5139 value = 3.1278 dict = 1.00 all = 3.4796 } Zip/UnZip{ index = 0.0951 value = 0.3197 dict = 1.00 all = 0.2874 } ---------------------------- total value len = 0.560537 GB avg = 0.117 KB (by entry num) total key len = 0.134444 GB avg = 0.028 KB total ukey len = 0.096031 GB avg = 0.020 KB total ukey num = 0.004801564 Billion total entry num = 0.004801564 Billion write speed all = 0.009009032 MB/sec (with seq num) write speed all = 0.008511091 MB/sec (without seq num) 2024/11/23-06:18:05.844591 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248218: 924651 keys, 38572427 bytes, temperature: kWarm 2024/11/23-06:18:05.844674 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313885844629, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248218, "file_size": 38572427, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3433383142, "table_properties": {"data_size": 34633328, "index_size": 1758912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25890228, "raw_average_key_size": 28, "raw_value_size": 108327538, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924651, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313882, "oldest_key_time": 0, "file_creation_time": 1732313882, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248218, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:18:05.905661 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248219.sst) /dev/shm/ToplingTemp/Topling-432069-aBq7kf 2024/11/23-06:18:06.817467 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248219.sst): first pass time = 0.91's, 147.209'MB/sec 2024/11/23-06:18:08.379910 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248219.sst): old prealloc_size = 147639500, real_size = 34221680 2024/11/23-06:18:08.380455 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248219.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929502 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0322 GB avg-val = 116.40 avg-zval = 34.66 2024/11/23-06:18:08.381257 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248219.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:18:09.444383 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248219.sst): second pass time = 1.56's, 69.274'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 36.116'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3388.268'MB/sec (index lex order gen) rebuild zvType time = 0.93's, 0.250'MB/sec write SST data time = 0.14's, 283.854'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35682.391'MB/sec zip my value time = 1.56's, unzip length = 0.108'GB zip my value throughput = 69.274'MB/sec zip pipeline throughput = 69.274'MB/sec entries = 929502 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.40 avg-zval = 34.66 usrkeys = 929502 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.40 avg-zval = 34.66 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929502 bytes = 3369456 } seq expand size = 6506514 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0322 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5128 value = 3.3578 dict = 1.00 all = 3.4913 } Zip/UnZip{ index = 0.0951 value = 0.2978 dict = 1.00 all = 0.2864 } ---------------------------- total value len = 0.675236 GB avg = 0.118 KB (by entry num) total key len = 0.160470 GB avg = 0.028 KB total ukey len = 0.114621 GB avg = 0.020 KB total ukey num = 0.005731066 Billion total entry num = 0.005731066 Billion write speed all = 0.010832729 MB/sec (with seq num) write speed all = 0.010238423 MB/sec (without seq num) 2024/11/23-06:18:09.479192 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248219: 929502 keys, 38444052 bytes, temperature: kWarm 2024/11/23-06:18:09.479257 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313889479224, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248219, "file_size": 38444052, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3433383143, "largest_seqno": 3831419167, "table_properties": {"data_size": 32220832, "index_size": 1768320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26026056, "raw_average_key_size": 28, "raw_value_size": 108191776, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929502, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313885, "oldest_key_time": 0, "file_creation_time": 1732313885, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248219, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:18:09.513970 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248220.sst) /dev/shm/ToplingTemp/Topling-432069-kjCCMe 2024/11/23-06:18:10.496417 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248220.sst): first pass time = 0.98's, 136.623'MB/sec 2024/11/23-06:18:12.037815 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248220.sst): old prealloc_size = 147639500, real_size = 33785360 2024/11/23-06:18:12.038508 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248220.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932925 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0318 GB avg-val = 115.87 avg-zval = 34.06 2024/11/23-06:18:12.039368 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248220.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:18:13.064528 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248220.sst): second pass time = 1.54's, 70.206'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 37.142'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2632.684'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.268'MB/sec write SST data time = 0.16's, 243.718'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 30204.836'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.206'MB/sec zip pipeline throughput = 70.206'MB/sec entries = 932925 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.87 avg-zval = 34.06 usrkeys = 932925 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.87 avg-zval = 34.06 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932925 bytes = 3498480 } seq expand size = 6530475 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5116 value = 3.4017 dict = 1.00 all = 3.5196 } Zip/UnZip{ index = 0.0951 value = 0.2940 dict = 1.00 all = 0.2841 } ---------------------------- total value len = 0.789862 GB avg = 0.119 KB (by entry num) total key len = 0.186592 GB avg = 0.028 KB total ukey len = 0.133280 GB avg = 0.020 KB total ukey num = 0.006663991 Billion total entry num = 0.006663991 Billion write speed all = 0.012656566 MB/sec (with seq num) write speed all = 0.011965549 MB/sec (without seq num) 2024/11/23-06:18:13.100135 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248220: 932925 keys, 38134320 bytes, temperature: kWarm 2024/11/23-06:18:13.100199 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313893100167, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248220, "file_size": 38134320, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3831419284, "table_properties": {"data_size": 31776896, "index_size": 1775040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26121900, "raw_average_key_size": 28, "raw_value_size": 108095838, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932925, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313889, "oldest_key_time": 0, "file_creation_time": 1732313889, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248220, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:18:13.137028 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248221.sst) /dev/shm/ToplingTemp/Topling-432069-cTMhvc 2024/11/23-06:18:13.975803 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248221.sst): first pass time = 0.84's, 160.023'MB/sec 2024/11/23-06:18:15.304171 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248221.sst): old prealloc_size = 147639500, real_size = 30729392 2024/11/23-06:18:15.304715 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248221.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927592 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1082 GB zip-val = 0.0287 GB avg-val = 116.69 avg-zval = 30.94 2024/11/23-06:18:15.305473 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248221.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:18:16.170939 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248221.sst): second pass time = 1.33's, 81.533'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 40.049'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3460.907'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.13's, 270.327'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31543.753'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.533'MB/sec zip pipeline throughput = 81.533'MB/sec entries = 927592 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.69 avg-zval = 30.94 usrkeys = 927592 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.69 avg-zval = 30.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927592 bytes = 3130640 } seq expand size = 6493144 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0287 GB dict = 1.08 MB all = 0.0347 GB } UnZip/Zip{ index = 10.3102 value = 3.7719 dict = 1.00 all = 3.8666 } Zip/UnZip{ index = 0.0970 value = 0.2651 dict = 1.00 all = 0.2586 } ---------------------------- total value len = 0.904600 GB avg = 0.119 KB (by entry num) total key len = 0.212564 GB avg = 0.028 KB total ukey len = 0.151832 GB avg = 0.020 KB total ukey num = 0.007591583 Billion total entry num = 0.007591583 Billion write speed all = 0.014479846 MB/sec (with seq num) write speed all = 0.013692675 MB/sec (without seq num) 2024/11/23-06:18:16.201906 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248221: 927592 keys, 34712358 bytes, temperature: kWarm 2024/11/23-06:18:16.201969 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313896201936, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248221, "file_size": 34712358, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3710787473, "largest_seqno": 3831416866, "table_properties": {"data_size": 28697952, "index_size": 1799360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25972576, "raw_average_key_size": 28, "raw_value_size": 108245208, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927592, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313893, "oldest_key_time": 0, "file_creation_time": 1732313893, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248221, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:18:16.239147 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248222.sst) /dev/shm/ToplingTemp/Topling-432069-VniAWc 2024/11/23-06:18:17.037978 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248222.sst): first pass time = 0.80's, 168.026'MB/sec 2024/11/23-06:18:18.372988 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248222.sst): old prealloc_size = 147639500, real_size = 32100192 2024/11/23-06:18:18.373503 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248222.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926902 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0301 GB avg-val = 116.80 avg-zval = 32.47 2024/11/23-06:18:18.375432 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248222.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:18:19.261827 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248222.sst): second pass time = 1.33's, 81.135'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 40.857'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3605.735'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.306'MB/sec write SST data time = 0.13's, 276.463'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27448.406'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.135'MB/sec zip pipeline throughput = 81.135'MB/sec entries = 926902 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.80 avg-zval = 32.47 usrkeys = 926902 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.80 avg-zval = 32.47 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926902 bytes = 3360032 } seq expand size = 6488314 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.4430 value = 3.5976 dict = 1.00 all = 3.6960 } Zip/UnZip{ index = 0.0958 value = 0.2780 dict = 1.00 all = 0.2706 } ---------------------------- total value len = 1.019353 GB avg = 0.120 KB (by entry num) total key len = 0.238518 GB avg = 0.028 KB total ukey len = 0.170370 GB avg = 0.020 KB total ukey num = 0.008518485 Billion total entry num = 0.008518485 Billion write speed all = 0.016302919 MB/sec (with seq num) write speed all = 0.015419673 MB/sec (without seq num) 2024/11/23-06:18:19.295081 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248222: 926902 keys, 36314424 bytes, temperature: kWarm 2024/11/23-06:18:19.295143 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313899295112, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248222, "file_size": 36314424, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3831419075, "table_properties": {"data_size": 30093136, "index_size": 1775168, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953256, "raw_average_key_size": 28, "raw_value_size": 108264512, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926902, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313896, "oldest_key_time": 0, "file_creation_time": 1732313896, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248222, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:18:19.330242 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248223.sst) /dev/shm/ToplingTemp/Topling-432069-9PR2oc 2024/11/23-06:18:20.127934 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248223.sst): first pass time = 0.80's, 168.265'MB/sec 2024/11/23-06:18:21.478638 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248223.sst): old prealloc_size = 147639500, real_size = 32075664 2024/11/23-06:18:21.479230 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248223.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928241 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0301 GB avg-val = 116.59 avg-zval = 32.40 2024/11/23-06:18:21.481110 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248223.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:18:22.376817 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248223.sst): second pass time = 1.35's, 80.171'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.281'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3084.508'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.302'MB/sec write SST data time = 0.13's, 280.987'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30477.588'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.171'MB/sec zip pipeline throughput = 80.171'MB/sec entries = 928241 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.59 avg-zval = 32.40 usrkeys = 928241 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.59 avg-zval = 32.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928241 bytes = 3248864 } seq expand size = 6497687 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0362 GB } UnZip/Zip{ index = 10.5126 value = 3.5983 dict = 1.00 all = 3.7099 } Zip/UnZip{ index = 0.0951 value = 0.2779 dict = 1.00 all = 0.2695 } ---------------------------- total value len = 1.134078 GB avg = 0.120 KB (by entry num) total key len = 0.264508 GB avg = 0.028 KB total ukey len = 0.188935 GB avg = 0.020 KB total ukey num = 0.009446726 Billion total entry num = 0.009446726 Billion write speed all = 0.018125962 MB/sec (with seq num) write speed all = 0.017146510 MB/sec (without seq num) 2024/11/23-06:18:22.409134 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248223: 928241 keys, 36178016 bytes, temperature: kWarm 2024/11/23-06:18:22.409197 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313902409165, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248223, "file_size": 36178016, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3610601794, "largest_seqno": 3831418511, "table_properties": {"data_size": 30077440, "index_size": 1765952, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25990748, "raw_average_key_size": 28, "raw_value_size": 108227042, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928241, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313899, "oldest_key_time": 0, "file_creation_time": 1732313899, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248223, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:18:22.443302 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248224.sst) /dev/shm/ToplingTemp/Topling-432069-ZRNS7e 2024/11/23-06:18:23.340311 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248224.sst): first pass time = 0.90's, 149.634'MB/sec 2024/11/23-06:18:24.693429 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248224.sst): old prealloc_size = 147639500, real_size = 31996752 2024/11/23-06:18:24.694025 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248224.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928427 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0300 GB avg-val = 116.56 avg-zval = 32.31 2024/11/23-06:18:24.694827 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248224.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:18:25.583807 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248224.sst): second pass time = 1.35's, 80.018'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.89's, 40.676'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3095.101'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.305'MB/sec write SST data time = 0.13's, 281.659'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31132.534'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.018'MB/sec zip pipeline throughput = 80.018'MB/sec entries = 928427 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.56 avg-zval = 32.31 usrkeys = 928427 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.56 avg-zval = 32.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928427 bytes = 3365568 } seq expand size = 6498989 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0300 GB dict = 1.08 MB all = 0.0362 GB } UnZip/Zip{ index = 10.5117 value = 3.6076 dict = 1.00 all = 3.7061 } Zip/UnZip{ index = 0.0951 value = 0.2772 dict = 1.00 all = 0.2698 } ---------------------------- total value len = 1.248799 GB avg = 0.120 KB (by entry num) total key len = 0.290504 GB avg = 0.028 KB total ukey len = 0.207503 GB avg = 0.020 KB total ukey num = 0.010375153 Billion total entry num = 0.010375153 Billion write speed all = 0.019948851 MB/sec (with seq num) write speed all = 0.018873183 MB/sec (without seq num) 2024/11/23-06:18:25.616923 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248224: 928427 keys, 36215694 bytes, temperature: kWarm 2024/11/23-06:18:25.616987 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313905616953, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248224, "file_size": 36215694, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3516470397, "largest_seqno": 3831418980, "table_properties": {"data_size": 29998016, "index_size": 1766464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25995956, "raw_average_key_size": 28, "raw_value_size": 108221862, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928427, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313902, "oldest_key_time": 0, "file_creation_time": 1732313902, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248224, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:18:25.656801 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248225.sst) /dev/shm/ToplingTemp/Topling-432069-kQrhye 2024/11/23-06:18:26.504715 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248225.sst): first pass time = 0.85's, 158.299'MB/sec 2024/11/23-06:18:27.824502 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248225.sst): old prealloc_size = 147639500, real_size = 31215168 2024/11/23-06:18:27.825102 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248225.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928330 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0292 GB avg-val = 116.58 avg-zval = 31.46 2024/11/23-06:18:27.827024 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248225.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-06:18:28.692505 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248225.sst): second pass time = 1.32's, 82.044'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 40.823'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3081.617'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.315'MB/sec write SST data time = 0.13's, 271.710'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32394.059'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.044'MB/sec zip pipeline throughput = 82.044'MB/sec entries = 928330 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.58 avg-zval = 31.46 usrkeys = 928330 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.58 avg-zval = 31.46 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928330 bytes = 3365216 } seq expand size = 6498310 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0354 GB } UnZip/Zip{ index = 10.4448 value = 3.7056 dict = 1.00 all = 3.7878 } Zip/UnZip{ index = 0.0957 value = 0.2699 dict = 1.00 all = 0.2640 } ---------------------------- total value len = 1.363522 GB avg = 0.121 KB (by entry num) total key len = 0.316498 GB avg = 0.028 KB total ukey len = 0.226070 GB avg = 0.020 KB total ukey num = 0.011303483 Billion total entry num = 0.011303483 Billion write speed all = 0.021771607 MB/sec (with seq num) write speed all = 0.020599739 MB/sec (without seq num) 2024/11/23-06:18:28.731020 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248225: 928330 keys, 35433758 bytes, temperature: kWarm 2024/11/23-06:18:28.731084 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313908731052, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248225, "file_size": 35433758, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3831416925, "table_properties": {"data_size": 29205296, "index_size": 1777600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25993240, "raw_average_key_size": 28, "raw_value_size": 108224496, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928330, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313905, "oldest_key_time": 0, "file_creation_time": 1732313905, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248225, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:18:28.767243 140025196639808 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248226.sst) /dev/shm/ToplingTemp/Topling-432069-Lc2zue 2024/11/23-06:18:28.981827 140025196639808 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248226.sst): first pass time = 0.21's, 249.647'MB/sec 2024/11/23-06:18:29.522383 140025196639808 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248226.sst): old prealloc_size = 147639500, real_size = 14281232 2024/11/23-06:18:29.522643 140025196639808 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248226.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 366858 prefix = 4 raw-key = 0.0059 GB zip-key = 0.0007 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0433 GB zip-val = 0.0135 GB avg-val = 118.00 avg-zval = 36.76 2024/11/23-06:18:29.523008 140025196639808 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248226.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.054 GB 2024/11/23-06:18:29.903985 140025196639808 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00/248226.sst): second pass time = 0.54's, 80.174'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.38's, 41.582'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2967.397'MB/sec (index lex order gen) rebuild zvType time = 0.33's, 0.278'MB/sec write SST data time = 0.05's, 307.005'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.437'MB, throughput = 14810.428'MB/sec zip my value time = 0.54's, unzip length = 0.043'GB zip my value throughput = 80.174'MB/sec zip pipeline throughput = 80.174'MB/sec entries = 366858 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 36.76 usrkeys = 366858 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 36.76 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 366858 bytes = 1238160 } seq expand size = 2568006 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0073 GB value = 0.0433 GB dict = 0.44 MB all = 0.0536 GB } __ZipSize{ index = 0.0007 GB value = 0.0135 GB dict = 0.44 MB all = 0.0159 GB } UnZip/Zip{ index = 10.4287 value = 3.2100 dict = 1.00 all = 3.3756 } Zip/UnZip{ index = 0.0959 value = 0.3115 dict = 1.00 all = 0.2962 } ---------------------------- total value len = 1.409379 GB avg = 0.121 KB (by entry num) total key len = 0.326770 GB avg = 0.028 KB total ukey len = 0.233407 GB avg = 0.020 KB total ukey num = 0.011670341 Billion total entry num = 0.011670341 Billion write speed all = 0.022498641 MB/sec (with seq num) write speed all = 0.021288759 MB/sec (without seq num) 2024/11/23-06:18:29.921227 140025196639808 [compaction_job.cc:1948] [default] [JOB 20880] Generated table #248226: 366858 keys, 15867025 bytes, temperature: kWarm 2024/11/23-06:18:29.921285 140025196639808 EVENT_LOG_v1 {"time_micros": 1732313909921255, "cf_name": "default", "job": 20880, "event": "table_file_creation", "file_number": 248226, "file_size": 15867025, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3681679274, "largest_seqno": 3764473211, "table_properties": {"data_size": 13485760, "index_size": 703552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 10272024, "raw_average_key_size": 28, "raw_value_size": 43289244, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 366858, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732313908, "oldest_key_time": 0, "file_creation_time": 1732313908, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 248226, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-06:18:29.936794 140025196639808 [compaction_job.cc:703] job-20880: subcompact[0], size: 0.459203 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-20880/att-00 [248214,248215,248216,248217,248218,248219,248220,248221,248222,248223,248224,248225,248226] 2024/11/23-06:18:30.019759 140025196639808 (Original Log Time 2024/11/23-06:17:45.832796) EVENT_LOG_v1 {"time_micros": 1732313865832746, "job": 20880, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [248209], "files_L3": [248172, 248173, 248174, 248175, 248176, 248154, 248155, 248156, 248157, 248158, 248159, 248160, 248161], "score": 0.999977, "input_data_size": 486997629, "oldest_snapshot_seqno": 3831438813} 2024/11/23-06:18:30.019763 140025196639808 (Original Log Time 2024/11/23-06:18:30.019068) [compaction_job.cc:2022] [default] [JOB 20880] Compacted 1@2 + 13@3 files to L3 => 459202935 bytes 2024/11/23-06:18:30.019764 140025196639808 (Original Log Time 2024/11/23-06:18:30.019679) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 11.0 rd, 10.4 wr, level 3, files in(1, 13) out(13 +0 blob) MB in(27.1, 437.3 +0.0 blob) out(437.9 +0.0 blob), read-write-amplify(33.3) write-amplify(16.2) OK, records in: 11734061, records dropped: 63720 output_compression: Snappy 2024/11/23-06:18:30.019770 140025196639808 (Original Log Time 2024/11/23-06:18:30.019738) EVENT_LOG_v1 {"time_micros": 1732313910019702, "cf": "default", "job": 20880, "event": "compaction_finished", "compaction_time_micros": 44104005, "compaction_time_cpu_micros": 19602699, "output_level": 3, "num_output_files": 13, "total_output_size": 459202935, "num_input_records": 11734061, "num_output_records": 11670341, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 310649044, "file_range_sync_nanos": 0, "file_fsync_nanos": 13461, "file_prepare_write_nanos": 33152, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-06:18:30.153906 140025196639808 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732313865,"executesMs":44305,"compactionJobId":20880,"attempt":0,"compactionInputRawBytes":1673222975,"compactionInputZipBytes":486997629,"compactionOutputRawBytes":0,"compactionOutputZipBytes":459202935,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-91e4fdd1e35237e9a351c89b78e1bf45-84934251a3fbc994-00"}}, response = 2024/11/23-06:18:30.153974 140025196639808 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-20880/att-00: olev 3, work 44.306 s, result 4.862 ms, install 0.719 ms, input{raw 1.558 GiB zip 464.437 MiB}