2024/11/23-12:51:27.747857 139957626406464 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00: fork to child time = 0.006450 sec 2024/11/23-12:51:27.748537 139957626406464 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00 2024/11/23-12:51:27.755306 139957626406464 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00 2024/11/23-12:51:27.756231 139957626406464 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-12:51:27.756441 139957626406464 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-12:51:27.808182 139957626406464 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-26327/att-00/MANIFEST-206347 2024/11/23-12:51:27.857708 139957626406464 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-26327/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 260188, last_sequence is 3982969852, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 260020 2024/11/23-12:51:27.857714 139957626406464 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-12:51:27.857813 139957626406464 [version_set.cc:5721] Creating manifest 260188 2024/11/23-12:51:27.890033 139957626406464 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-12:51:27.912529 139957626406464 [compaction_job.cc:2437] [default] [JOB 26327] Compacting 1@2 + 14@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-12:51:27.912542 139957626406464 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 19727 Base level 2, inputs: [260121(24MB)], [260186(20MB) 260082(33MB) 260083(36MB) 260084(36MB) 260085(36MB) 260086(32MB) 260087(33MB) 260088(34MB) 260089(34MB) 260090(33MB) 260091(34MB) 260092(34MB) 260093(12MB) 260026(13MB)] 2024/11/23-12:51:27.920297 139957626406464 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-12:51:27.920323 139957626406464 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-12:51:27.920338 139957626406464 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-12:51:27.920344 139957626406464 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-12:51:27.920356 139957626406464 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-12:51:28.339859 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260189.sst) /dev/shm/ToplingTemp/Topling-53139-H0EASh 2024/11/23-12:51:29.369510 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260189.sst): first pass time = 1.03's, 130.359'MB/sec 2024/11/23-12:51:30.811770 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260189.sst): old prealloc_size = 147639500, real_size = 32913296 2024/11/23-12:51:30.812324 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260189.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930778 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.20 avg-zval = 33.20 2024/11/23-12:51:30.813150 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260189.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:51:31.670212 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260189.sst): second pass time = 1.44's, 75.044'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.264'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3325.840'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 296.763'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36882.118'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.044'MB/sec zip pipeline throughput = 75.044'MB/sec entries = 930778 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.20 avg-zval = 33.20 usrkeys = 930778 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.20 avg-zval = 33.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930778 bytes = 3374080 } seq expand size = 6515446 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.4437 value = 3.5004 dict = 1.00 all = 3.6139 } Zip/UnZip{ index = 0.0958 value = 0.2857 dict = 1.00 all = 0.2767 } ---------------------------- total value len = 8.963291 GB avg = 0.168 KB (by entry num) total key len = 1.466883 GB avg = 0.028 KB total ukey len = 1.016494 GB avg = 0.015 KB total ukey num = 0.066481213 Billion total entry num = 0.053288942 Billion write speed all = 0.735583540 MB/sec (with seq num) write speed all = 0.705518102 MB/sec (without seq num) 2024/11/23-12:51:31.715402 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260189: 930778 keys, 37139030 bytes, temperature: kWarm 2024/11/23-12:51:31.715497 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337491715440, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260189, "file_size": 37139030, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3610601794, "largest_seqno": 3982684776, "table_properties": {"data_size": 30897984, "index_size": 1782464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26061784, "raw_average_key_size": 28, "raw_value_size": 108156008, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930778, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337488, "oldest_key_time": 0, "file_creation_time": 1732337488, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260189, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:51:31.750050 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260190.sst) /dev/shm/ToplingTemp/Topling-53139-m0jQdl 2024/11/23-12:51:32.652155 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260190.sst): first pass time = 0.90's, 148.790'MB/sec 2024/11/23-12:51:34.085275 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260190.sst): old prealloc_size = 147639500, real_size = 32965648 2024/11/23-12:51:34.085783 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260190.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934628 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0310 GB avg-val = 115.61 avg-zval = 33.12 2024/11/23-12:51:34.086627 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260190.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:51:34.942046 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260190.sst): second pass time = 1.43's, 75.436'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.562'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3666.170'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.320'MB/sec write SST data time = 0.13's, 296.737'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 29774.169'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.436'MB/sec zip pipeline throughput = 75.436'MB/sec entries = 934628 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.61 avg-zval = 33.12 usrkeys = 934628 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.61 avg-zval = 33.12 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934628 bytes = 3504864 } seq expand size = 6542396 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5137 value = 3.4906 dict = 1.00 all = 3.5963 } Zip/UnZip{ index = 0.0951 value = 0.2865 dict = 1.00 all = 0.2781 } ---------------------------- total value len = 9.077882 GB avg = 0.167 KB (by entry num) total key len = 1.493052 GB avg = 0.028 KB total ukey len = 1.035186 GB avg = 0.015 KB total ukey num = 0.067415841 Billion total entry num = 0.054223570 Billion write speed all = 0.745338613 MB/sec (with seq num) write speed all = 0.714752919 MB/sec (without seq num) 2024/11/23-12:51:34.987652 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260190: 934628 keys, 37320648 bytes, temperature: kWarm 2024/11/23-12:51:34.987720 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337494987684, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260190, "file_size": 37320648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3982685172, "table_properties": {"data_size": 30953856, "index_size": 1777920, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26169584, "raw_average_key_size": 28, "raw_value_size": 108048256, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934628, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337491, "oldest_key_time": 0, "file_creation_time": 1732337491, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260190, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:51:35.024297 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260191.sst) /dev/shm/ToplingTemp/Topling-53139-YkYCAi 2024/11/23-12:51:36.065412 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260191.sst): first pass time = 1.04's, 128.922'MB/sec 2024/11/23-12:51:37.435536 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260191.sst): old prealloc_size = 147639500, real_size = 30473040 2024/11/23-12:51:37.436099 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260191.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929512 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0285 GB avg-val = 116.40 avg-zval = 30.62 2024/11/23-12:51:37.436883 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260191.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:51:38.252470 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260191.sst): second pass time = 1.37's, 79.010'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 42.469'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3287.741'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.335'MB/sec write SST data time = 0.12's, 281.500'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 37106.665'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.010'MB/sec zip pipeline throughput = 79.010'MB/sec entries = 929512 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.40 avg-zval = 30.62 usrkeys = 929512 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.40 avg-zval = 30.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929512 bytes = 3369488 } seq expand size = 6506584 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0285 GB dict = 1.08 MB all = 0.0347 GB } UnZip/Zip{ index = 10.4445 value = 3.8015 dict = 1.00 all = 3.8687 } Zip/UnZip{ index = 0.0957 value = 0.2631 dict = 1.00 all = 0.2585 } ---------------------------- total value len = 9.192580 GB avg = 0.167 KB (by entry num) total key len = 1.519079 GB avg = 0.028 KB total ukey len = 1.053776 GB avg = 0.015 KB total ukey num = 0.068345353 Billion total entry num = 0.055153082 Billion write speed all = 0.755084600 MB/sec (with seq num) write speed all = 0.723981859 MB/sec (without seq num) 2024/11/23-12:51:38.298032 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260191: 929512 keys, 34693566 bytes, temperature: kWarm 2024/11/23-12:51:38.298107 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337498298072, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260191, "file_size": 34693566, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3683977965, "largest_seqno": 3982684821, "table_properties": {"data_size": 28460544, "index_size": 1779904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26026336, "raw_average_key_size": 28, "raw_value_size": 108191428, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929512, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337495, "oldest_key_time": 0, "file_creation_time": 1732337495, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260191, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:51:38.333525 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260192.sst) /dev/shm/ToplingTemp/Topling-53139-StFSEi 2024/11/23-12:51:39.348628 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260192.sst): first pass time = 1.02's, 132.226'MB/sec 2024/11/23-12:51:40.845246 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260192.sst): old prealloc_size = 147639500, real_size = 35480672 2024/11/23-12:51:40.845761 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260192.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927449 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0335 GB avg-val = 116.72 avg-zval = 36.09 2024/11/23-12:51:40.846480 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260192.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:51:41.670901 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260192.sst): second pass time = 1.50's, 72.372'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.570'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3597.972'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 299.753'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32839.308'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.372'MB/sec zip pipeline throughput = 72.372'MB/sec entries = 927449 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.72 avg-zval = 36.09 usrkeys = 927449 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.34 avg-zval = 36.09 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 744452 bytes = 2791712 } seq expand size = 5211164 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.4454 value = 3.2339 dict = 1.00 all = 3.4174 } Zip/UnZip{ index = 0.0957 value = 0.3092 dict = 1.00 all = 0.2926 } ---------------------------- total value len = 9.306040 GB avg = 0.166 KB (by entry num) total key len = 1.545047 GB avg = 0.028 KB total ukey len = 1.072325 GB avg = 0.015 KB total ukey num = 0.069272802 Billion total entry num = 0.056080531 Billion write speed all = 0.764728925 MB/sec (with seq num) write speed all = 0.733110783 MB/sec (without seq num) 2024/11/23-12:51:41.722180 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260192: 927449 keys, 39275140 bytes, temperature: kWarm 2024/11/23-12:51:41.722250 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337501722215, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260192, "file_size": 39275140, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3982685538, "table_properties": {"data_size": 33472848, "index_size": 1775808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25968572, "raw_average_key_size": 28, "raw_value_size": 108249194, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927449, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337498, "oldest_key_time": 0, "file_creation_time": 1732337498, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260192, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:51:41.758953 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260193.sst) /dev/shm/ToplingTemp/Topling-53139-2BQ8gm 2024/11/23-12:51:42.601290 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260193.sst): first pass time = 0.84's, 159.350'MB/sec 2024/11/23-12:51:44.068093 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260193.sst): old prealloc_size = 147639500, real_size = 33186640 2024/11/23-12:51:44.068597 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260193.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928649 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0312 GB avg-val = 116.53 avg-zval = 33.57 2024/11/23-12:51:44.069398 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260193.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:51:44.925076 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260193.sst): second pass time = 1.47's, 73.841'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.650'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3692.392'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.12's, 299.839'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 34068.552'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.841'MB/sec zip pipeline throughput = 73.841'MB/sec entries = 928649 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.53 avg-zval = 33.57 usrkeys = 928649 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.53 avg-zval = 33.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928649 bytes = 3366368 } seq expand size = 6500543 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.4453 value = 3.4711 dict = 1.00 all = 3.5881 } Zip/UnZip{ index = 0.0957 value = 0.2881 dict = 1.00 all = 0.2787 } ---------------------------- total value len = 9.420756 GB avg = 0.165 KB (by entry num) total key len = 1.571050 GB avg = 0.028 KB total ukey len = 1.090898 GB avg = 0.016 KB total ukey num = 0.070201451 Billion total entry num = 0.057009180 Billion write speed all = 0.774468414 MB/sec (with seq num) write speed all = 0.742334070 MB/sec (without seq num) 2024/11/23-12:51:44.972303 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260193: 928649 keys, 37406100 bytes, temperature: kWarm 2024/11/23-12:51:44.972371 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337504972336, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260193, "file_size": 37406100, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3512539342, "largest_seqno": 3982685400, "table_properties": {"data_size": 31176192, "index_size": 1778112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26002172, "raw_average_key_size": 28, "raw_value_size": 108215586, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928649, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337501, "oldest_key_time": 0, "file_creation_time": 1732337501, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260193, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:51:45.007404 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260194.sst) /dev/shm/ToplingTemp/Topling-53139-hu334k 2024/11/23-12:51:45.997716 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260194.sst): first pass time = 0.99's, 135.537'MB/sec 2024/11/23-12:51:47.519833 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260194.sst): old prealloc_size = 147639500, real_size = 32601376 2024/11/23-12:51:47.520381 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260194.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930829 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0306 GB avg-val = 116.19 avg-zval = 32.86 2024/11/23-12:51:47.521188 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260194.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:51:48.370821 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260194.sst): second pass time = 1.52's, 71.124'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.277'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3381.770'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.321'MB/sec write SST data time = 0.12's, 295.557'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 37726.948'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.124'MB/sec zip pipeline throughput = 71.124'MB/sec entries = 930829 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.19 avg-zval = 32.86 usrkeys = 930829 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.19 avg-zval = 32.86 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930829 bytes = 3374272 } seq expand size = 6515803 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.4432 value = 3.5361 dict = 1.00 all = 3.6445 } Zip/UnZip{ index = 0.0958 value = 0.2828 dict = 1.00 all = 0.2744 } ---------------------------- total value len = 9.535427 GB avg = 0.165 KB (by entry num) total key len = 1.597113 GB avg = 0.028 KB total ukey len = 1.109515 GB avg = 0.016 KB total ukey num = 0.071132280 Billion total entry num = 0.057940009 Billion write speed all = 0.784193930 MB/sec (with seq num) write speed all = 0.751542833 MB/sec (without seq num) 2024/11/23-12:51:48.417031 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260194: 930829 keys, 36827418 bytes, temperature: kWarm 2024/11/23-12:51:48.417110 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337508417074, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260194, "file_size": 36827418, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3604582272, "largest_seqno": 3982685631, "table_properties": {"data_size": 30585808, "index_size": 1782656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26063212, "raw_average_key_size": 28, "raw_value_size": 108154562, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930829, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337504, "oldest_key_time": 0, "file_creation_time": 1732337504, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260194, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:51:48.453771 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260195.sst) /dev/shm/ToplingTemp/Topling-53139-UfS7wi 2024/11/23-12:51:49.454292 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260195.sst): first pass time = 1.00's, 134.152'MB/sec 2024/11/23-12:51:50.899406 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260195.sst): old prealloc_size = 147639500, real_size = 32233008 2024/11/23-12:51:50.899902 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260195.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931055 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0302 GB avg-val = 116.16 avg-zval = 32.47 2024/11/23-12:51:50.900715 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260195.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:51:51.742347 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260195.sst): second pass time = 1.44's, 74.877'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.254'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3700.417'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 291.370'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 37060.170'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 74.877'MB/sec zip pipeline throughput = 74.877'MB/sec entries = 931055 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.16 avg-zval = 32.47 usrkeys = 931055 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.16 avg-zval = 32.47 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931055 bytes = 3375088 } seq expand size = 6517385 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.5129 value = 3.5777 dict = 1.00 all = 3.6813 } Zip/UnZip{ index = 0.0951 value = 0.2795 dict = 1.00 all = 0.2716 } ---------------------------- total value len = 9.650092 GB avg = 0.164 KB (by entry num) total key len = 1.623182 GB avg = 0.028 KB total ukey len = 1.128136 GB avg = 0.016 KB total ukey num = 0.072063335 Billion total entry num = 0.058871064 Billion write speed all = 0.793918995 MB/sec (with seq num) write speed all = 0.760751095 MB/sec (without seq num) 2024/11/23-12:51:51.788000 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260195: 931055 keys, 36459624 bytes, temperature: kWarm 2024/11/23-12:51:51.788078 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337511788034, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260195, "file_size": 36459624, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3485447251, "largest_seqno": 3982684983, "table_properties": {"data_size": 30228768, "index_size": 1771264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26069540, "raw_average_key_size": 28, "raw_value_size": 108148326, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337508, "oldest_key_time": 0, "file_creation_time": 1732337508, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260195, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:51:51.822311 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260196.sst) /dev/shm/ToplingTemp/Topling-53139-zveX1i 2024/11/23-12:51:52.869271 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260196.sst): first pass time = 1.05's, 128.202'MB/sec 2024/11/23-12:51:54.313616 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260196.sst): old prealloc_size = 147639500, real_size = 31866240 2024/11/23-12:51:54.314246 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260196.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930801 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0299 GB avg-val = 116.20 avg-zval = 32.07 2024/11/23-12:51:54.315086 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260196.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:51:55.184820 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260196.sst): second pass time = 1.44's, 74.923'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 41.562'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2931.562'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.312'MB/sec write SST data time = 0.12's, 290.454'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 37130.138'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 74.923'MB/sec zip pipeline throughput = 74.923'MB/sec entries = 930801 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.20 avg-zval = 32.07 usrkeys = 930801 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.20 avg-zval = 32.07 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930801 bytes = 3490512 } seq expand size = 6515607 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0299 GB dict = 1.08 MB all = 0.0362 GB } UnZip/Zip{ index = 10.4455 value = 3.6232 dict = 1.00 all = 3.7068 } Zip/UnZip{ index = 0.0957 value = 0.2760 dict = 1.00 all = 0.2698 } ---------------------------- total value len = 9.764763 GB avg = 0.163 KB (by entry num) total key len = 1.649245 GB avg = 0.028 KB total ukey len = 1.146752 GB avg = 0.016 KB total ukey num = 0.072994136 Billion total entry num = 0.059801865 Billion write speed all = 0.803635296 MB/sec (with seq num) write speed all = 0.769951150 MB/sec (without seq num) 2024/11/23-12:51:55.229598 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260196: 930801 keys, 36208290 bytes, temperature: kWarm 2024/11/23-12:51:55.229669 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337515229632, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260196, "file_size": 36208290, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3411274482, "largest_seqno": 3982684997, "table_properties": {"data_size": 29851120, "index_size": 1782208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26062428, "raw_average_key_size": 28, "raw_value_size": 108155350, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930801, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337511, "oldest_key_time": 0, "file_creation_time": 1732337511, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260196, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:51:55.267639 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260197.sst) /dev/shm/ToplingTemp/Topling-53139-ycRdZk 2024/11/23-12:51:56.099487 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260197.sst): first pass time = 0.83's, 161.356'MB/sec 2024/11/23-12:51:57.581846 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260197.sst): old prealloc_size = 147639500, real_size = 33137136 2024/11/23-12:51:57.582466 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260197.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927738 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.67 avg-zval = 33.57 2024/11/23-12:51:57.583274 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260197.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:51:58.449660 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260197.sst): second pass time = 1.48's, 73.058'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 43.046'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2956.802'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.13's, 290.714'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 38022.456'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.058'MB/sec zip pipeline throughput = 73.058'MB/sec entries = 927738 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.67 avg-zval = 33.57 usrkeys = 927738 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.67 avg-zval = 33.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927738 bytes = 3363072 } seq expand size = 6494166 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5115 value = 3.4760 dict = 1.00 all = 3.5931 } Zip/UnZip{ index = 0.0951 value = 0.2877 dict = 1.00 all = 0.2783 } ---------------------------- total value len = 9.879499 GB avg = 0.163 KB (by entry num) total key len = 1.675221 GB avg = 0.028 KB total ukey len = 1.165307 GB avg = 0.016 KB total ukey num = 0.073921874 Billion total entry num = 0.060729603 Billion write speed all = 0.813355546 MB/sec (with seq num) write speed all = 0.779156702 MB/sec (without seq num) 2024/11/23-12:51:58.496904 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260197: 927738 keys, 37354384 bytes, temperature: kWarm 2024/11/23-12:51:58.497483 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337518496938, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260197, "file_size": 37354384, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3525889795, "largest_seqno": 3982685007, "table_properties": {"data_size": 31139808, "index_size": 1765184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25976664, "raw_average_key_size": 28, "raw_value_size": 108241068, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927738, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337515, "oldest_key_time": 0, "file_creation_time": 1732337515, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260197, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:51:58.536884 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260198.sst) /dev/shm/ToplingTemp/Topling-53139-dJyoOj 2024/11/23-12:51:59.385537 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260198.sst): first pass time = 0.85's, 158.160'MB/sec 2024/11/23-12:52:00.825993 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260198.sst): old prealloc_size = 147639500, real_size = 31732144 2024/11/23-12:52:00.826548 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260198.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930963 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0297 GB avg-val = 116.17 avg-zval = 31.93 2024/11/23-12:52:00.828083 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260198.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:52:01.687569 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260198.sst): second pass time = 1.44's, 75.125'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 41.736'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3354.939'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.319'MB/sec write SST data time = 0.13's, 272.558'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35940.024'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.125'MB/sec zip pipeline throughput = 75.125'MB/sec entries = 930963 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.17 avg-zval = 31.93 usrkeys = 930963 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.17 avg-zval = 31.93 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930963 bytes = 3374752 } seq expand size = 6516741 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0297 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.5126 value = 3.6380 dict = 1.00 all = 3.7326 } Zip/UnZip{ index = 0.0951 value = 0.2749 dict = 1.00 all = 0.2679 } ---------------------------- total value len = 9.994166 GB avg = 0.162 KB (by entry num) total key len = 1.701288 GB avg = 0.028 KB total ukey len = 1.183926 GB avg = 0.016 KB total ukey num = 0.074852837 Billion total entry num = 0.061660566 Billion write speed all = 0.823074481 MB/sec (with seq num) write speed all = 0.788359293 MB/sec (without seq num) 2024/11/23-12:52:01.734397 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260198: 930963 keys, 35958648 bytes, temperature: kWarm 2024/11/23-12:52:01.734472 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337521734434, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260198, "file_size": 35958648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3596991077, "largest_seqno": 3982685017, "table_properties": {"data_size": 29728096, "index_size": 1771136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26066964, "raw_average_key_size": 28, "raw_value_size": 108150802, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930963, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337518, "oldest_key_time": 0, "file_creation_time": 1732337518, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260198, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:52:01.776177 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260199.sst) /dev/shm/ToplingTemp/Topling-53139-jaK0Fk 2024/11/23-12:52:02.797188 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260199.sst): first pass time = 1.02's, 131.462'MB/sec 2024/11/23-12:52:04.226988 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260199.sst): old prealloc_size = 147639500, real_size = 30238416 2024/11/23-12:52:04.227570 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260199.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931088 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0282 GB avg-val = 116.15 avg-zval = 30.32 2024/11/23-12:52:04.229219 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260199.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:52:05.098152 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260199.sst): second pass time = 1.43's, 75.705'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 39.696'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3165.815'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.325'MB/sec write SST data time = 0.15's, 223.736'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 39502.381'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.705'MB/sec zip pipeline throughput = 75.705'MB/sec entries = 931088 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.15 avg-zval = 30.32 usrkeys = 931088 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.15 avg-zval = 30.32 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931088 bytes = 3491600 } seq expand size = 6517616 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0282 GB dict = 1.08 MB all = 0.0346 GB } UnZip/Zip{ index = 10.5133 value = 3.8304 dict = 1.00 all = 3.8812 } Zip/UnZip{ index = 0.0951 value = 0.2611 dict = 1.00 all = 0.2577 } ---------------------------- total value len = 10.108831 GB avg = 0.162 KB (by entry num) total key len = 1.727359 GB avg = 0.028 KB total ukey len = 1.202548 GB avg = 0.016 KB total ukey num = 0.075783925 Billion total entry num = 0.062591654 Billion write speed all = 0.832778929 MB/sec (with seq num) write speed all = 0.797547990 MB/sec (without seq num) 2024/11/23-12:52:05.140777 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260199: 931088 keys, 34581392 bytes, temperature: kWarm 2024/11/23-12:52:05.140846 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337525140811, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260199, "file_size": 34581392, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 3982685282, "table_properties": {"data_size": 28234176, "index_size": 1771264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26070464, "raw_average_key_size": 28, "raw_value_size": 108147276, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931088, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337521, "oldest_key_time": 0, "file_creation_time": 1732337521, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260199, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:52:05.177668 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260200.sst) /dev/shm/ToplingTemp/Topling-53139-qfuYRk 2024/11/23-12:52:05.897013 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260200.sst): first pass time = 0.72's, 186.592'MB/sec 2024/11/23-12:52:07.231167 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260200.sst): old prealloc_size = 147639500, real_size = 32971872 2024/11/23-12:52:07.231774 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260200.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927908 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0310 GB avg-val = 116.65 avg-zval = 33.37 2024/11/23-12:52:07.233369 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260200.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:52:08.076722 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260200.sst): second pass time = 1.33's, 81.179'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 44.120'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3048.816'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.323'MB/sec write SST data time = 0.13's, 293.721'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27727.630'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.179'MB/sec zip pipeline throughput = 81.179'MB/sec entries = 927908 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.65 avg-zval = 33.37 usrkeys = 927908 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.65 avg-zval = 33.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927908 bytes = 3479664 } seq expand size = 6495356 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.4423 value = 3.4957 dict = 1.00 all = 3.5978 } Zip/UnZip{ index = 0.0958 value = 0.2861 dict = 1.00 all = 0.2779 } ---------------------------- total value len = 10.223563 GB avg = 0.161 KB (by entry num) total key len = 1.753340 GB avg = 0.028 KB total ukey len = 1.221106 GB avg = 0.016 KB total ukey num = 0.076711833 Billion total entry num = 0.063519562 Billion write speed all = 0.842502760 MB/sec (with seq num) write speed all = 0.806757020 MB/sec (without seq num) 2024/11/23-12:52:08.121795 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260200: 927908 keys, 37305268 bytes, temperature: kWarm 2024/11/23-12:52:08.121866 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337528121829, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260200, "file_size": 37305268, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3406445153, "largest_seqno": 3982685043, "table_properties": {"data_size": 30962512, "index_size": 1777216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25981424, "raw_average_key_size": 28, "raw_value_size": 108236344, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927908, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337525, "oldest_key_time": 0, "file_creation_time": 1732337525, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260200, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:52:08.159200 139957626406464 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260201.sst) /dev/shm/ToplingTemp/Topling-53139-ya9Bdj 2024/11/23-12:52:08.599163 139957626406464 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260201.sst): first pass time = 0.44's, 135.472'MB/sec 2024/11/23-12:52:09.196892 139957626406464 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260201.sst): old prealloc_size = 147639500, real_size = 15677872 2024/11/23-12:52:09.197231 139957626406464 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260201.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 409170 prefix = 4 raw-key = 0.0065 GB zip-key = 0.0008 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0481 GB zip-val = 0.0148 GB avg-val = 117.65 avg-zval = 36.14 2024/11/23-12:52:09.197619 139957626406464 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260201.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0010 GB, waited 0.000 sec, Key+Value = 0.060 GB 2024/11/23-12:52:09.574329 139957626406464 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00/260201.sst): second pass time = 0.60's, 80.642'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.38's, 46.503'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2502.445'MB/sec (index lex order gen) rebuild zvType time = 0.32's, 0.320'MB/sec write SST data time = 0.06's, 306.880'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.490'MB, throughput = 16667.664'MB/sec zip my value time = 0.60's, unzip length = 0.048'GB zip my value throughput = 80.642'MB/sec zip pipeline throughput = 80.642'MB/sec entries = 409170 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.65 avg-zval = 36.14 usrkeys = 409170 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.65 avg-zval = 36.14 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 409170 bytes = 1483264 } seq expand size = 2864190 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0082 GB value = 0.0481 GB dict = 0.49 MB all = 0.0596 GB } __ZipSize{ index = 0.0008 GB value = 0.0148 GB dict = 0.49 MB all = 0.0176 GB } UnZip/Zip{ index = 10.3652 value = 3.2558 dict = 1.00 all = 3.3956 } Zip/UnZip{ index = 0.0965 value = 0.3071 dict = 1.00 all = 0.2945 } ---------------------------- total value len = 10.274567 GB avg = 0.161 KB (by entry num) total key len = 1.764797 GB avg = 0.028 KB total ukey len = 1.229289 GB avg = 0.016 KB total ukey num = 0.077121003 Billion total entry num = 0.063928732 Billion write speed all = 0.846807329 MB/sec (with seq num) write speed all = 0.810835118 MB/sec (without seq num) 2024/11/23-12:52:09.599386 139957626406464 [compaction_job.cc:1948] [default] [JOB 26327] Generated table #260201: 409170 keys, 17551259 bytes, temperature: kWarm 2024/11/23-12:52:09.599450 139957626406464 EVENT_LOG_v1 {"time_micros": 1732337529599419, "cf_name": "default", "job": 26327, "event": "table_file_creation", "file_number": 260201, "file_size": 17551259, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3664789595, "largest_seqno": 3982685458, "table_properties": {"data_size": 14785888, "index_size": 789504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 11456760, "raw_average_key_size": 28, "raw_value_size": 48140408, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 409170, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732337528, "oldest_key_time": 0, "file_creation_time": 1732337528, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 260201, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:52:09.618763 139957626406464 [compaction_job.cc:703] job-26327: subcompact[0], size: 0.458081 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-26327/att-00 [260189,260190,260191,260192,260193,260194,260195,260196,260197,260198,260199,260200,260201] 2024/11/23-12:52:09.698482 139957626406464 (Original Log Time 2024/11/23-12:51:27.912584) EVENT_LOG_v1 {"time_micros": 1732337487912552, "job": 26327, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [260121], "files_L3": [260186, 260082, 260083, 260084, 260085, 260086, 260087, 260088, 260089, 260090, 260091, 260092, 260093, 260026], "score": 0.997814, "input_data_size": 474211439, "oldest_snapshot_seqno": 3982969709} 2024/11/23-12:52:09.698489 139957626406464 (Original Log Time 2024/11/23-12:52:09.697302) [compaction_job.cc:2022] [default] [JOB 26327] Compacted 1@2 + 14@3 files to L3 => 458080767 bytes 2024/11/23-12:52:09.698491 139957626406464 (Original Log Time 2024/11/23-12:52:09.698410) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 11.4 rd, 11.0 wr, level 3, files in(1, 14) out(13 +0 blob) MB in(24.9, 427.3 +0.0 blob) out(436.9 +0.0 blob), read-write-amplify(35.7) write-amplify(17.5) OK, records in: 11624419, records dropped: 53851 output_compression: Snappy 2024/11/23-12:52:09.698501 139957626406464 (Original Log Time 2024/11/23-12:52:09.698458) EVENT_LOG_v1 {"time_micros": 1732337529698434, "cf": "default", "job": 26327, "event": "compaction_finished", "compaction_time_micros": 41706188, "compaction_time_cpu_micros": 17511245, "output_level": 3, "num_output_files": 13, "total_output_size": 458080767, "num_input_records": 11624419, "num_output_records": 11570568, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 312037223, "file_range_sync_nanos": 0, "file_fsync_nanos": 14359, "file_prepare_write_nanos": 36320, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-12:52:09.826808 139957626406464 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732337487,"executesMs":41943,"compactionJobId":26327,"attempt":0,"compactionInputRawBytes":1677857354,"compactionInputZipBytes":474211439,"compactionOutputRawBytes":0,"compactionOutputZipBytes":458080767,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-24793e6cae2d16dd2d8e5329ddc00f1d-d2372676cd62d62d-00"}}, response = 2024/11/23-12:52:09.826900 139957626406464 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-26327/att-00: olev 3, work 41.943 s, result 5.981 ms, install 1.217 ms, input{raw 1.563 GiB zip 452.243 MiB}