2024/11/24-06:12:17.025242 139957341054528 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00: fork to child time = 0.011238 sec 2024/11/24-06:12:17.026049 139957341054528 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00 2024/11/24-06:12:17.031736 139957341054528 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00 2024/11/24-06:12:17.032668 139957341054528 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-06:12:17.032926 139957341054528 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-06:12:17.085135 139957341054528 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-42676/att-00/MANIFEST-206347 2024/11/24-06:12:17.110723 139957341054528 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-42676/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 294591, last_sequence is 4474011155, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 294431 2024/11/24-06:12:17.110730 139957341054528 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-06:12:17.110858 139957341054528 [version_set.cc:5721] Creating manifest 294591 2024/11/24-06:12:17.147634 139957341054528 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-06:12:17.168954 139957341054528 [compaction_job.cc:2437] [default] [JOB 42676] Compacting 1@2 + 17@3 files to L3, score 0.84, subcompactions 1 : 1 2024/11/24-06:12:17.168970 139957341054528 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 31991 Base level 2, inputs: [294566(27MB)], [294588(35MB) 294589(3065KB) 294510(37MB) 294511(6574KB) 294486(33MB) 294487(33MB) 294488(35MB) 294489(33MB) 294490(35MB) 294491(33MB) 294492(34MB) 294493(34MB) 294494(33MB) 294495(35MB) 294496(33MB) 294497(35MB) 294498(33MB)] 2024/11/24-06:12:17.175686 139957341054528 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-06:12:17.175716 139957341054528 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-06:12:17.175736 139957341054528 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-06:12:17.175741 139957341054528 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-06:12:17.175752 139957341054528 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-06:12:17.882885 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294592.sst) /dev/shm/ToplingTemp/Topling-286812-wCj0PA 2024/11/24-06:12:18.479095 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294592.sst): first pass time = 0.60's, 225.131'MB/sec 2024/11/24-06:12:20.326955 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294592.sst): old prealloc_size = 147639500, real_size = 33412416 2024/11/24-06:12:20.327585 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294592.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931109 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0314 GB avg-val = 116.15 avg-zval = 33.73 2024/11/24-06:12:20.328442 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294592.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:21.203646 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294592.sst): second pass time = 1.85's, 58.598'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.801'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2949.778'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.14's, 259.788'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 35637.589'MB/sec zip my value time = 1.85's, unzip length = 0.108'GB zip my value throughput = 58.598'MB/sec zip pipeline throughput = 65.019'MB/sec entries = 931109 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.15 avg-zval = 33.73 usrkeys = 931109 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.15 avg-zval = 33.73 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931109 bytes = 3258896 } seq expand size = 6517763 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5139 value = 3.4433 dict = 1.00 all = 3.5770 } Zip/UnZip{ index = 0.0951 value = 0.2904 dict = 1.00 all = 0.2796 } ---------------------------- total value len = 2251.440238 GB avg = 0.200 KB (by entry num) total key len = 306.909034 GB avg = 0.027 KB total ukey len = 237.373350 GB avg = 0.015 KB total ukey num = 15.739592272 Billion total entry num = 11.280906220 Billion write speed all = 33.386180671 MB/sec (with seq num) write speed all = 32.208463844 MB/sec (without seq num) 2024/11/24-06:12:21.248820 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294592: 931109 keys, 37522904 bytes, temperature: kWarm 2024/11/24-06:12:21.248912 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399941248856, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294592, "file_size": 37522904, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178518, "largest_seqno": 4473935386, "table_properties": {"data_size": 31408240, "index_size": 1771200, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26071052, "raw_average_key_size": 28, "raw_value_size": 108146730, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931109, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399937, "oldest_key_time": 0, "file_creation_time": 1732399937, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294592, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:21.287580 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294593.sst) /dev/shm/ToplingTemp/Topling-286812-fkQtyA 2024/11/24-06:12:22.495207 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294593.sst): first pass time = 1.21's, 111.145'MB/sec 2024/11/24-06:12:24.297484 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294593.sst): old prealloc_size = 147639500, real_size = 35767360 2024/11/24-06:12:24.298225 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294593.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926362 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0338 GB avg-val = 116.89 avg-zval = 36.46 2024/11/24-06:12:24.299094 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294593.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:25.210187 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294593.sst): second pass time = 1.80's, 60.152'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 43.678'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2451.232'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.305'MB/sec write SST data time = 0.15's, 261.698'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34356.582'MB/sec zip my value time = 1.80's, unzip length = 0.108'GB zip my value throughput = 60.152'MB/sec zip pipeline throughput = 67.017'MB/sec entries = 926362 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.89 avg-zval = 36.46 usrkeys = 926362 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.89 avg-zval = 36.46 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926362 bytes = 3242288 } seq expand size = 6484534 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0338 GB dict = 1.08 MB all = 0.0399 GB } UnZip/Zip{ index = 10.5142 value = 3.2061 dict = 1.00 all = 3.3669 } Zip/UnZip{ index = 0.0951 value = 0.3119 dict = 1.00 all = 0.2970 } ---------------------------- total value len = 2251.555002 GB avg = 0.200 KB (by entry num) total key len = 306.934972 GB avg = 0.027 KB total ukey len = 237.391877 GB avg = 0.015 KB total ukey num = 15.740518634 Billion total entry num = 11.281832582 Billion write speed all = 33.386271223 MB/sec (with seq num) write speed all = 32.208519264 MB/sec (without seq num) 2024/11/24-06:12:25.263857 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294593: 926362 keys, 39864162 bytes, temperature: kWarm 2024/11/24-06:12:25.263921 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399945263888, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294593, "file_size": 39864162, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4268980586, "largest_seqno": 4473935640, "table_properties": {"data_size": 33773488, "index_size": 1762112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25938136, "raw_average_key_size": 28, "raw_value_size": 108279592, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926362, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399941, "oldest_key_time": 0, "file_creation_time": 1732399941, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294593, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:25.299947 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294594.sst) /dev/shm/ToplingTemp/Topling-286812-faFOTA 2024/11/24-06:12:26.240835 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294594.sst): first pass time = 0.94's, 142.665'MB/sec 2024/11/24-06:12:27.692040 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294594.sst): old prealloc_size = 147639500, real_size = 31508720 2024/11/24-06:12:27.692589 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294594.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919459 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1085 GB zip-val = 0.0295 GB avg-val = 117.97 avg-zval = 32.09 2024/11/24-06:12:27.693395 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294594.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:28.521100 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294594.sst): second pass time = 1.45's, 74.787'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 42.925'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3353.563'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.328'MB/sec write SST data time = 0.13's, 276.618'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 35938.268'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.787'MB/sec zip pipeline throughput = 74.787'MB/sec entries = 919459 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.97 avg-zval = 32.09 usrkeys = 919459 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.97 avg-zval = 32.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919459 bytes = 3218128 } seq expand size = 6436213 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.09 MB all = 0.0356 GB } UnZip/Zip{ index = 10.3767 value = 3.6762 dict = 1.00 all = 3.7716 } Zip/UnZip{ index = 0.0964 value = 0.2720 dict = 1.00 all = 0.2651 } ---------------------------- total value len = 2251.669911 GB avg = 0.200 KB (by entry num) total key len = 306.960717 GB avg = 0.027 KB total ukey len = 237.410266 GB avg = 0.015 KB total ukey num = 15.741438093 Billion total entry num = 11.282752041 Billion write speed all = 33.386664182 MB/sec (with seq num) write speed all = 32.208867123 MB/sec (without seq num) 2024/11/24-06:12:28.567306 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294594: 919459 keys, 35586164 bytes, temperature: kWarm 2024/11/24-06:12:28.567391 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399948567343, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294594, "file_size": 35586164, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4323857544, "largest_seqno": 4473935242, "table_properties": {"data_size": 29506528, "index_size": 1772160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25744852, "raw_average_key_size": 28, "raw_value_size": 108472918, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 919459, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399945, "oldest_key_time": 0, "file_creation_time": 1732399945, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294594, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:28.610698 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294595.sst) /dev/shm/ToplingTemp/Topling-286812-RsXSMB 2024/11/24-06:12:29.492719 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294595.sst): first pass time = 0.88's, 152.178'MB/sec 2024/11/24-06:12:30.951102 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294595.sst): old prealloc_size = 147639500, real_size = 33155088 2024/11/24-06:12:30.951686 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294595.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927690 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0312 GB avg-val = 116.68 avg-zval = 33.59 2024/11/24-06:12:30.952475 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294595.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:31.819015 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294595.sst): second pass time = 1.46's, 74.265'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 42.928'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3145.863'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.13's, 289.622'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32673.222'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.265'MB/sec zip pipeline throughput = 78.656'MB/sec entries = 927690 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.68 avg-zval = 33.59 usrkeys = 927690 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.68 avg-zval = 33.59 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927690 bytes = 3246928 } seq expand size = 6493830 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5129 value = 3.4740 dict = 1.00 all = 3.6026 } Zip/UnZip{ index = 0.0951 value = 0.2879 dict = 1.00 all = 0.2776 } ---------------------------- total value len = 2251.784647 GB avg = 0.200 KB (by entry num) total key len = 306.986692 GB avg = 0.027 KB total ukey len = 237.428820 GB avg = 0.015 KB total ukey num = 15.742365783 Billion total entry num = 11.283679731 Billion write speed all = 33.387063534 MB/sec (with seq num) write speed all = 32.209220320 MB/sec (without seq num) 2024/11/24-06:12:31.865906 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294595: 927690 keys, 37255900 bytes, temperature: kWarm 2024/11/24-06:12:31.866000 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399951865948, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294595, "file_size": 37255900, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178599, "largest_seqno": 4473935514, "table_properties": {"data_size": 31158144, "index_size": 1764864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25975320, "raw_average_key_size": 28, "raw_value_size": 108242508, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927690, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399948, "oldest_key_time": 0, "file_creation_time": 1732399948, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294595, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:31.904321 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294596.sst) /dev/shm/ToplingTemp/Topling-286812-mRjA9A 2024/11/24-06:12:32.920078 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294596.sst): first pass time = 1.02's, 132.143'MB/sec 2024/11/24-06:12:34.249091 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294596.sst): old prealloc_size = 147639500, real_size = 30790688 2024/11/24-06:12:34.249663 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294596.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926452 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0288 GB avg-val = 116.87 avg-zval = 31.05 2024/11/24-06:12:34.250440 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294596.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:35.070483 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294596.sst): second pass time = 1.33's, 81.562'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 42.333'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3283.014'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 274.376'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32854.580'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.562'MB/sec zip pipeline throughput = 81.562'MB/sec entries = 926452 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.87 avg-zval = 31.05 usrkeys = 926452 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.87 avg-zval = 31.05 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926452 bytes = 3126784 } seq expand size = 6485164 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0288 GB dict = 1.08 MB all = 0.0348 GB } UnZip/Zip{ index = 10.3108 value = 3.7646 dict = 1.00 all = 3.8601 } Zip/UnZip{ index = 0.0970 value = 0.2656 dict = 1.00 all = 0.2591 } ---------------------------- total value len = 2251.899410 GB avg = 0.200 KB (by entry num) total key len = 307.012632 GB avg = 0.027 KB total ukey len = 237.447349 GB avg = 0.015 KB total ukey num = 15.743292235 Billion total entry num = 11.284606183 Billion write speed all = 33.387482949 MB/sec (with seq num) write speed all = 32.209593001 MB/sec (without seq num) 2024/11/24-06:12:35.114898 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294596: 926452 keys, 34770364 bytes, temperature: kWarm 2024/11/24-06:12:35.114970 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399955114933, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294596, "file_size": 34770364, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4365238843, "largest_seqno": 4473935256, "table_properties": {"data_size": 28761808, "index_size": 1797056, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25940656, "raw_average_key_size": 28, "raw_value_size": 108277072, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926452, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399951, "oldest_key_time": 0, "file_creation_time": 1732399951, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294596, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:35.151727 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294597.sst) /dev/shm/ToplingTemp/Topling-286812-Hs3Twz 2024/11/24-06:12:36.095472 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294597.sst): first pass time = 0.94's, 142.224'MB/sec 2024/11/24-06:12:37.428839 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294597.sst): old prealloc_size = 147639500, real_size = 32090304 2024/11/24-06:12:37.429425 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294597.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920681 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0301 GB avg-val = 117.78 avg-zval = 32.68 2024/11/24-06:12:37.430206 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294597.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:38.250710 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294597.sst): second pass time = 1.33's, 81.375'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 44.019'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3146.216'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.331'MB/sec write SST data time = 0.13's, 285.574'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 34680.720'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.375'MB/sec zip pipeline throughput = 81.375'MB/sec entries = 920681 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.78 avg-zval = 32.68 usrkeys = 920681 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.78 avg-zval = 32.68 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920681 bytes = 3222400 } seq expand size = 6444767 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0362 GB } UnZip/Zip{ index = 10.3774 value = 3.6043 dict = 1.00 all = 3.7100 } Zip/UnZip{ index = 0.0964 value = 0.2774 dict = 1.00 all = 0.2695 } ---------------------------- total value len = 2252.014293 GB avg = 0.200 KB (by entry num) total key len = 307.038412 GB avg = 0.027 KB total ukey len = 237.465762 GB avg = 0.015 KB total ukey num = 15.744212916 Billion total entry num = 11.285526864 Billion write speed all = 33.387932844 MB/sec (with seq num) write speed all = 32.209995673 MB/sec (without seq num) 2024/11/24-06:12:38.296353 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294597: 920681 keys, 36177162 bytes, temperature: kWarm 2024/11/24-06:12:38.296454 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399958296394, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294597, "file_size": 36177162, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178645, "largest_seqno": 4473935429, "table_properties": {"data_size": 30085552, "index_size": 1774400, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25779068, "raw_average_key_size": 28, "raw_value_size": 108438686, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920681, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399955, "oldest_key_time": 0, "file_creation_time": 1732399955, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294597, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:38.333224 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294598.sst) /dev/shm/ToplingTemp/Topling-286812-mDktIA 2024/11/24-06:12:39.355693 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294598.sst): first pass time = 1.02's, 131.273'MB/sec 2024/11/24-06:12:40.702956 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294598.sst): old prealloc_size = 147639500, real_size = 32603680 2024/11/24-06:12:40.703551 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294598.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927799 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0306 GB avg-val = 116.66 avg-zval = 32.98 2024/11/24-06:12:40.704362 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294598.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:41.545180 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294598.sst): second pass time = 1.35's, 80.385'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.583'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3254.565'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 289.972'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34641.226'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.385'MB/sec zip pipeline throughput = 80.385'MB/sec entries = 927799 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.66 avg-zval = 32.98 usrkeys = 927799 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.66 avg-zval = 32.98 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927799 bytes = 3247312 } seq expand size = 6494593 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.4437 value = 3.5378 dict = 1.00 all = 3.6566 } Zip/UnZip{ index = 0.0958 value = 0.2827 dict = 1.00 all = 0.2735 } ---------------------------- total value len = 2252.129027 GB avg = 0.200 KB (by entry num) total key len = 307.064390 GB avg = 0.027 KB total ukey len = 237.484318 GB avg = 0.015 KB total ukey num = 15.745140715 Billion total entry num = 11.286454663 Billion write speed all = 33.388333590 MB/sec (with seq num) write speed all = 32.210350212 MB/sec (without seq num) 2024/11/24-06:12:41.591331 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294598: 927799 keys, 36705164 bytes, temperature: kWarm 2024/11/24-06:12:41.591412 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399961591372, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294598, "file_size": 36705164, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178655, "largest_seqno": 4473935287, "table_properties": {"data_size": 30594768, "index_size": 1776768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25978372, "raw_average_key_size": 28, "raw_value_size": 108239414, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927799, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399958, "oldest_key_time": 0, "file_creation_time": 1732399958, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294598, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:41.626643 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294599.sst) /dev/shm/ToplingTemp/Topling-286812-aG6sWx 2024/11/24-06:12:42.450492 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294599.sst): first pass time = 0.82's, 162.923'MB/sec 2024/11/24-06:12:43.775952 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294599.sst): old prealloc_size = 147639500, real_size = 31333088 2024/11/24-06:12:43.776532 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294599.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0293 GB avg-val = 118.00 avg-zval = 31.92 2024/11/24-06:12:43.777311 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294599.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:44.590025 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294599.sst): second pass time = 1.32's, 81.890'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.499'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3312.509'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 281.793'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 28971.456'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.890'MB/sec zip pipeline throughput = 81.890'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.91 avg-val = 118.00 avg-zval = 31.92 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.91 avg-val = 125.00 avg-zval = 31.92 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3217568 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0293 GB dict = 1.09 MB all = 0.0354 GB } UnZip/Zip{ index = 10.4458 value = 3.6969 dict = 1.00 all = 3.7904 } Zip/UnZip{ index = 0.0957 value = 0.2705 dict = 1.00 all = 0.2638 } ---------------------------- total value len = 2252.243939 GB avg = 0.200 KB (by entry num) total key len = 307.090130 GB avg = 0.027 KB total ukey len = 237.502704 GB avg = 0.015 KB total ukey num = 15.746060015 Billion total entry num = 11.287373963 Billion write speed all = 33.388842263 MB/sec (with seq num) write speed all = 32.210809732 MB/sec (without seq num) 2024/11/24-06:12:44.635422 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294599: 919300 keys, 35409778 bytes, temperature: kWarm 2024/11/24-06:12:44.635493 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399964635457, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294599, "file_size": 35409778, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4295709319, "largest_seqno": 4447355331, "table_properties": {"data_size": 29342928, "index_size": 1760128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399961, "oldest_key_time": 0, "file_creation_time": 1732399961, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294599, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:44.672462 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294600.sst) /dev/shm/ToplingTemp/Topling-286812-g6rqUy 2024/11/24-06:12:45.524802 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294600.sst): first pass time = 0.85's, 157.477'MB/sec 2024/11/24-06:12:46.875073 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294600.sst): old prealloc_size = 147639500, real_size = 33453680 2024/11/24-06:12:46.875632 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294600.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927485 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0315 GB avg-val = 116.71 avg-zval = 33.92 2024/11/24-06:12:46.876434 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294600.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:47.734668 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294600.sst): second pass time = 1.35's, 80.219'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.690'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3291.730'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 296.522'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33046.119'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.219'MB/sec zip pipeline throughput = 80.219'MB/sec entries = 927485 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.71 avg-zval = 33.92 usrkeys = 927485 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.71 avg-zval = 33.92 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927485 bytes = 3246208 } seq expand size = 6492395 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5113 value = 3.4412 dict = 1.00 all = 3.5740 } Zip/UnZip{ index = 0.0951 value = 0.2906 dict = 1.00 all = 0.2798 } ---------------------------- total value len = 2252.358680 GB avg = 0.200 KB (by entry num) total key len = 307.116100 GB avg = 0.027 KB total ukey len = 237.521254 GB avg = 0.015 KB total ukey num = 15.746987500 Billion total entry num = 11.288301448 Billion write speed all = 33.389308161 MB/sec (with seq num) write speed all = 32.211227162 MB/sec (without seq num) 2024/11/24-06:12:47.780830 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294600: 927485 keys, 37554414 bytes, temperature: kWarm 2024/11/24-06:12:47.780913 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399967780867, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294600, "file_size": 37554414, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178675, "largest_seqno": 4473935113, "table_properties": {"data_size": 31456864, "index_size": 1764736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25969580, "raw_average_key_size": 28, "raw_value_size": 108248182, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927485, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399964, "oldest_key_time": 0, "file_creation_time": 1732399964, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294600, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:47.816651 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294601.sst) /dev/shm/ToplingTemp/Topling-286812-KKYQdA 2024/11/24-06:12:48.745715 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294601.sst): first pass time = 0.93's, 144.471'MB/sec 2024/11/24-06:12:50.055761 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294601.sst): old prealloc_size = 147639500, real_size = 31129312 2024/11/24-06:12:50.056329 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294601.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922138 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0291 GB avg-val = 117.55 avg-zval = 31.59 2024/11/24-06:12:50.057128 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294601.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:50.870459 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294601.sst): second pass time = 1.31's, 82.794'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.224'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3229.818'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.335'MB/sec write SST data time = 0.13's, 280.256'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 31597.352'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.794'MB/sec zip pipeline throughput = 82.794'MB/sec entries = 922138 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.55 avg-zval = 31.59 usrkeys = 922138 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.55 avg-zval = 31.59 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922138 bytes = 3227504 } seq expand size = 6454966 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0291 GB dict = 1.08 MB all = 0.0352 GB } UnZip/Zip{ index = 10.4431 value = 3.7209 dict = 1.00 all = 3.8115 } Zip/UnZip{ index = 0.0958 value = 0.2688 dict = 1.00 all = 0.2624 } ---------------------------- total value len = 2252.473533 GB avg = 0.200 KB (by entry num) total key len = 307.141920 GB avg = 0.027 KB total ukey len = 237.539697 GB avg = 0.015 KB total ukey num = 15.747909638 Billion total entry num = 11.289223586 Billion write speed all = 33.389777394 MB/sec (with seq num) write speed all = 32.211648353 MB/sec (without seq num) 2024/11/24-06:12:50.914067 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294601: 922138 keys, 35213602 bytes, temperature: kWarm 2024/11/24-06:12:50.914139 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399970914101, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294601, "file_size": 35213602, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4339710307, "largest_seqno": 4473934600, "table_properties": {"data_size": 29132560, "index_size": 1766016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25819864, "raw_average_key_size": 28, "raw_value_size": 108397892, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922138, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399967, "oldest_key_time": 0, "file_creation_time": 1732399967, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294601, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:50.951402 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294602.sst) /dev/shm/ToplingTemp/Topling-286812-DIbm7y 2024/11/24-06:12:51.824709 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294602.sst): first pass time = 0.87's, 153.696'MB/sec 2024/11/24-06:12:53.174352 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294602.sst): old prealloc_size = 147639500, real_size = 32976032 2024/11/24-06:12:53.174930 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294602.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924811 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0310 GB avg-val = 117.13 avg-zval = 33.50 2024/11/24-06:12:53.175756 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294602.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:54.021011 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294602.sst): second pass time = 1.35's, 80.312'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.784'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3244.023'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.13's, 289.930'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35121.068'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.312'MB/sec zip pipeline throughput = 80.312'MB/sec entries = 924811 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.13 avg-zval = 33.50 usrkeys = 924811 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.13 avg-zval = 33.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924811 bytes = 3236848 } seq expand size = 6473677 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5130 value = 3.4960 dict = 1.00 all = 3.6208 } Zip/UnZip{ index = 0.0951 value = 0.2860 dict = 1.00 all = 0.2762 } ---------------------------- total value len = 2252.588330 GB avg = 0.200 KB (by entry num) total key len = 307.167814 GB avg = 0.027 KB total ukey len = 237.558193 GB avg = 0.015 KB total ukey num = 15.748834449 Billion total entry num = 11.290148397 Billion write speed all = 33.390240414 MB/sec (with seq num) write speed all = 32.212063281 MB/sec (without seq num) 2024/11/24-06:12:54.066450 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294602: 924811 keys, 37068358 bytes, temperature: kWarm 2024/11/24-06:12:54.066522 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399974066484, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294602, "file_size": 37068358, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178686, "largest_seqno": 4473935355, "table_properties": {"data_size": 30985296, "index_size": 1759360, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25894708, "raw_average_key_size": 28, "raw_value_size": 108323130, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924811, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399970, "oldest_key_time": 0, "file_creation_time": 1732399970, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294602, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:54.107690 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294603.sst) /dev/shm/ToplingTemp/Topling-286812-JrxLHB 2024/11/24-06:12:55.129934 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294603.sst): first pass time = 1.02's, 131.302'MB/sec 2024/11/24-06:12:56.444283 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294603.sst): old prealloc_size = 147639500, real_size = 31424960 2024/11/24-06:12:56.444879 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294603.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927718 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0294 GB avg-val = 116.68 avg-zval = 31.71 2024/11/24-06:12:56.445709 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294603.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:12:57.269811 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294603.sst): second pass time = 1.31's, 82.406'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 43.036'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3100.813'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.332'MB/sec write SST data time = 0.13's, 282.576'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34585.516'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.406'MB/sec zip pipeline throughput = 82.406'MB/sec entries = 927718 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.68 avg-zval = 31.71 usrkeys = 927718 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.68 avg-zval = 31.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927718 bytes = 3247024 } seq expand size = 6494026 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.08 MB all = 0.0355 GB } UnZip/Zip{ index = 10.4431 value = 3.6797 dict = 1.00 all = 3.7780 } Zip/UnZip{ index = 0.0958 value = 0.2718 dict = 1.00 all = 0.2647 } ---------------------------- total value len = 2252.703065 GB avg = 0.200 KB (by entry num) total key len = 307.193791 GB avg = 0.027 KB total ukey len = 237.576747 GB avg = 0.015 KB total ukey num = 15.749762167 Billion total entry num = 11.291076115 Billion write speed all = 33.390660851 MB/sec (with seq num) write speed all = 32.212436839 MB/sec (without seq num) 2024/11/24-06:12:57.314393 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294603: 927718 keys, 35526020 bytes, temperature: kWarm 2024/11/24-06:12:57.314464 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399977314428, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294603, "file_size": 35526020, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178700, "largest_seqno": 4473934749, "table_properties": {"data_size": 29416112, "index_size": 1776704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25976104, "raw_average_key_size": 28, "raw_value_size": 108241636, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927718, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399974, "oldest_key_time": 0, "file_creation_time": 1732399974, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294603, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:12:57.354140 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294604.sst) /dev/shm/ToplingTemp/Topling-286812-cZzZFz 2024/11/24-06:12:58.354346 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294604.sst): first pass time = 1.00's, 134.196'MB/sec 2024/11/24-06:12:59.681325 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294604.sst): old prealloc_size = 147639500, real_size = 31339056 2024/11/24-06:12:59.681870 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294604.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0293 GB avg-val = 118.00 avg-zval = 31.93 2024/11/24-06:12:59.682653 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294604.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:13:00.494408 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294604.sst): second pass time = 1.33's, 81.797'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.559'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3442.823'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.335'MB/sec write SST data time = 0.13's, 280.066'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 31405.870'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.797'MB/sec zip pipeline throughput = 81.797'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.91 avg-val = 118.00 avg-zval = 31.93 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.91 avg-val = 125.00 avg-zval = 31.93 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3217568 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0293 GB dict = 1.09 MB all = 0.0354 GB } UnZip/Zip{ index = 10.4462 value = 3.6961 dict = 1.00 all = 3.7898 } Zip/UnZip{ index = 0.0957 value = 0.2706 dict = 1.00 all = 0.2639 } ---------------------------- total value len = 2252.817978 GB avg = 0.200 KB (by entry num) total key len = 307.219531 GB avg = 0.027 KB total ukey len = 237.595133 GB avg = 0.015 KB total ukey num = 15.750681467 Billion total entry num = 11.291995415 Billion write speed all = 33.391091039 MB/sec (with seq num) write speed all = 32.212820657 MB/sec (without seq num) 2024/11/24-06:13:00.540816 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294604: 919300 keys, 35415794 bytes, temperature: kWarm 2024/11/24-06:13:00.540886 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399980540851, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294604, "file_size": 35415794, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4271495973, "largest_seqno": 4423983037, "table_properties": {"data_size": 29348960, "index_size": 1760064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399977, "oldest_key_time": 0, "file_creation_time": 1732399977, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294604, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:13:00.579650 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294605.sst) /dev/shm/ToplingTemp/Topling-286812-8pEUQx 2024/11/24-06:13:01.468766 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294605.sst): first pass time = 0.89's, 150.963'MB/sec 2024/11/24-06:13:02.821376 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294605.sst): old prealloc_size = 147639500, real_size = 33072976 2024/11/24-06:13:02.821968 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294605.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927747 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0311 GB avg-val = 116.67 avg-zval = 33.50 2024/11/24-06:13:02.822770 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294605.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:13:03.701611 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294605.sst): second pass time = 1.35's, 80.071'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.233'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3009.630'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.309'MB/sec write SST data time = 0.13's, 289.232'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31004.133'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.071'MB/sec zip pipeline throughput = 80.071'MB/sec entries = 927747 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.67 avg-zval = 33.50 usrkeys = 927747 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.67 avg-zval = 33.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927747 bytes = 3247136 } seq expand size = 6494229 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5131 value = 3.4831 dict = 1.00 all = 3.6105 } Zip/UnZip{ index = 0.0951 value = 0.2871 dict = 1.00 all = 0.2770 } ---------------------------- total value len = 2252.932713 GB avg = 0.199 KB (by entry num) total key len = 307.245508 GB avg = 0.027 KB total ukey len = 237.613688 GB avg = 0.015 KB total ukey num = 15.751609214 Billion total entry num = 11.292923162 Billion write speed all = 33.391529532 MB/sec (with seq num) write speed all = 32.213211635 MB/sec (without seq num) 2024/11/24-06:13:03.747967 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294605: 927747 keys, 37174276 bytes, temperature: kWarm 2024/11/24-06:13:03.748053 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399983748015, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294605, "file_size": 37174276, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178719, "largest_seqno": 4473935469, "table_properties": {"data_size": 31075904, "index_size": 1764928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25976916, "raw_average_key_size": 28, "raw_value_size": 108240822, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927747, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399980, "oldest_key_time": 0, "file_creation_time": 1732399980, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294605, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:13:03.787687 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294606.sst) /dev/shm/ToplingTemp/Topling-286812-kVklZx 2024/11/24-06:13:04.687668 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294606.sst): first pass time = 0.90's, 149.140'MB/sec 2024/11/24-06:13:06.009131 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294606.sst): old prealloc_size = 147639500, real_size = 31536480 2024/11/24-06:13:06.009697 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294606.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1085 GB zip-val = 0.0295 GB avg-val = 118.00 avg-zval = 32.14 2024/11/24-06:13:06.010487 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294606.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-06:13:06.827162 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294606.sst): second pass time = 1.32's, 82.146'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.537'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3257.240'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.332'MB/sec write SST data time = 0.13's, 283.652'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 27847.903'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.146'MB/sec zip pipeline throughput = 82.146'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 32.14 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 32.14 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3217568 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.09 MB all = 0.0356 GB } UnZip/Zip{ index = 10.4436 value = 3.6715 dict = 1.00 all = 3.7688 } Zip/UnZip{ index = 0.0958 value = 0.2724 dict = 1.00 all = 0.2653 } ---------------------------- total value len = 2253.047625 GB avg = 0.199 KB (by entry num) total key len = 307.271248 GB avg = 0.027 KB total ukey len = 237.632074 GB avg = 0.015 KB total ukey num = 15.752528514 Billion total entry num = 11.293842462 Billion write speed all = 33.392002777 MB/sec (with seq num) write speed all = 32.213636996 MB/sec (without seq num) 2024/11/24-06:13:06.871968 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294606: 919300 keys, 35613170 bytes, temperature: kWarm 2024/11/24-06:13:06.872062 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399986872023, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294606, "file_size": 35613170, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4314162540, "largest_seqno": 4465404001, "table_properties": {"data_size": 29545936, "index_size": 1760512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399983, "oldest_key_time": 0, "file_creation_time": 1732399983, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294606, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:13:06.910526 139957341054528 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294607.sst) /dev/shm/ToplingTemp/Topling-286812-OjP7AA 2024/11/24-06:13:07.128890 139957341054528 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294607.sst): first pass time = 0.22's, 246.658'MB/sec 2024/11/24-06:13:07.639233 139957341054528 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294607.sst): old prealloc_size = 147639500, real_size = 13394128 2024/11/24-06:13:07.639512 139957341054528 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294607.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 377547 prefix = 4 raw-key = 0.0060 GB zip-key = 0.0007 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0433 GB zip-val = 0.0126 GB avg-val = 114.63 avg-zval = 33.32 2024/11/24-06:13:07.639857 139957341054528 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294607.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0009 GB, waited 0.000 sec, Key+Value = 0.054 GB 2024/11/24-06:13:07.982113 139957341054528 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00/294607.sst): second pass time = 0.51's, 84.933'MB/sec, value only(80.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.34's, 43.932'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2819.526'MB/sec (index lex order gen) rebuild zvType time = 0.29's, 0.324'MB/sec write SST data time = 0.05's, 292.052'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.438'MB, throughput = 14693.776'MB/sec zip my value time = 0.51's, unzip length = 0.043'GB zip my value throughput = 84.933'MB/sec zip pipeline throughput = 84.933'MB/sec entries = 377547 avg-key = 28.00 avg-zkey = 1.90 avg-val = 114.63 avg-zval = 33.32 usrkeys = 377547 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.63 avg-zval = 33.32 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 377547 bytes = 1321424 } seq expand size = 2642829 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0076 GB value = 0.0433 GB dict = 0.44 MB all = 0.0539 GB } __ZipSize{ index = 0.0007 GB value = 0.0126 GB dict = 0.44 MB all = 0.0151 GB } UnZip/Zip{ index = 10.4995 value = 3.4403 dict = 1.00 all = 3.5753 } Zip/UnZip{ index = 0.0952 value = 0.2907 dict = 1.00 all = 0.2797 } ---------------------------- total value len = 2253.093548 GB avg = 0.199 KB (by entry num) total key len = 307.281820 GB avg = 0.027 KB total ukey len = 237.639625 GB avg = 0.015 KB total ukey num = 15.752906061 Billion total entry num = 11.294220009 Billion write speed all = 33.392236591 MB/sec (with seq num) write speed all = 32.213849168 MB/sec (without seq num) 2024/11/24-06:13:08.005588 139957341054528 [compaction_job.cc:1948] [default] [JOB 42676] Generated table #294607: 377547 keys, 15062219 bytes, temperature: kWarm 2024/11/24-06:13:08.005655 139957341054528 EVENT_LOG_v1 {"time_micros": 1732399988005620, "cf_name": "default", "job": 42676, "event": "table_file_creation", "file_number": 294607, "file_size": 15062219, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178730, "largest_seqno": 4473934651, "table_properties": {"data_size": 12580416, "index_size": 719168, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 10571316, "raw_average_key_size": 28, "raw_value_size": 43279938, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 377547, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732399986, "oldest_key_time": 0, "file_creation_time": 1732399986, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 294607, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-06:13:08.022712 139957341054528 [compaction_job.cc:703] job-42676: subcompact[0], size: 0.561919 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-42676/att-00 [294592,294593,294594,294595,294596,294597,294598,294599,294600,294601,294602,294603,294604,294605,294606,294607] 2024/11/24-06:13:08.115685 139957341054528 (Original Log Time 2024/11/24-06:12:17.169062) EVENT_LOG_v1 {"time_micros": 1732399937169015, "job": 42676, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [294566], "files_L3": [294588, 294589, 294510, 294511, 294486, 294487, 294488, 294489, 294490, 294491, 294492, 294493, 294494, 294495, 294496, 294497, 294498], "score": 0.839223, "input_data_size": 583903943, "oldest_snapshot_seqno": 4474010767} 2024/11/24-06:13:08.115690 139957341054528 (Original Log Time 2024/11/24-06:13:08.114348) [compaction_job.cc:2022] [default] [JOB 42676] Compacted 1@2 + 17@3 files to L3 => 561919451 bytes 2024/11/24-06:13:08.115691 139957341054528 (Original Log Time 2024/11/24-06:13:08.115610) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 16 0 0 0] max score 0.02, MB/sec: 11.5 rd, 11.0 wr, level 3, files in(1, 17) out(16 +0 blob) MB in(27.2, 529.7 +0.0 blob) out(535.9 +0.0 blob), read-write-amplify(40.2) write-amplify(19.7) OK, records in: 14318952, records dropped: 74054 output_compression: Snappy 2024/11/24-06:13:08.115696 139957341054528 (Original Log Time 2024/11/24-06:13:08.115664) EVENT_LOG_v1 {"time_micros": 1732399988115636, "cf": "default", "job": 42676, "event": "compaction_finished", "compaction_time_micros": 50853660, "compaction_time_cpu_micros": 21534104, "output_level": 3, "num_output_files": 16, "total_output_size": 561919451, "num_input_records": 14318952, "num_output_records": 14244898, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 392678621, "file_range_sync_nanos": 0, "file_fsync_nanos": 11568, "file_prepare_write_nanos": 38983, "lsm_state": [0, 0, 0, 16, 0, 0, 0]} 2024/11/24-06:13:08.243119 139957341054528 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732399936,"executesMs":51082,"compactionJobId":42676,"attempt":0,"compactionInputRawBytes":2077633532,"compactionInputZipBytes":583903943,"compactionOutputRawBytes":0,"compactionOutputZipBytes":561919451,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-8d4965a0358767a763a6ee2383d30a88-e5739990ed201a89-00"}}, response = 2024/11/24-06:13:08.243202 139957341054528 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-42676/att-00: olev 3, work 51.083 s, result 6.033 ms, install 1.366 ms, input{raw 1.935 GiB zip 556.854 MiB}