2024/11/23-00:39:03.291428 140024718255680 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00: fork to child time = 0.009685 sec 2024/11/23-00:39:03.292346 140024718255680 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00 2024/11/23-00:39:03.299670 140024718255680 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00 2024/11/23-00:39:03.301295 140024718255680 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-00:39:03.301576 140024718255680 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-00:39:03.356864 140024718255680 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15433/att-00/MANIFEST-206347 2024/11/23-00:39:03.373437 140024718255680 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15433/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 236751, last_sequence is 3680998573, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 236608 2024/11/23-00:39:03.373445 140024718255680 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-00:39:03.373542 140024718255680 [version_set.cc:5721] Creating manifest 236751 2024/11/23-00:39:03.404785 140024718255680 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-00:39:03.433470 140024718255680 [compaction_job.cc:2437] [default] [JOB 15433] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-00:39:03.433489 140024718255680 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 11562 Base level 2, inputs: [236745(27MB)], [236683(36MB) 236684(34MB) 236685(37MB) 236686(35MB) 236687(35MB) 236688(38MB) 236689(35MB) 236690(35MB) 236691(36MB) 236692(36MB) 236693(35MB) 236712(36MB)] 2024/11/23-00:39:03.441400 140024718255680 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:39:03.441432 140024718255680 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:39:03.441453 140024718255680 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-00:39:03.441459 140024718255680 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-00:39:03.441470 140024718255680 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-00:39:03.927879 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236752.sst) /dev/shm/ToplingTemp/Topling-353485-oYqZD4 2024/11/23-00:39:04.516503 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236752.sst): first pass time = 0.59's, 228.038'MB/sec 2024/11/23-00:39:06.608321 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236752.sst): old prealloc_size = 147639500, real_size = 34075072 2024/11/23-00:39:06.608963 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236752.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933083 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0321 GB avg-val = 115.84 avg-zval = 34.37 2024/11/23-00:39:06.609842 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236752.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:08.294895 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236752.sst): second pass time = 2.09's, 51.776'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.69's, 22.644'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2876.901'MB/sec (index lex order gen) rebuild zvType time = 1.37's, 0.170'MB/sec write SST data time = 0.32's, 120.657'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33494.919'MB/sec zip my value time = 2.09's, unzip length = 0.108'GB zip my value throughput = 51.776'MB/sec zip pipeline throughput = 67.501'MB/sec entries = 933083 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.84 avg-zval = 34.37 usrkeys = 933083 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.84 avg-zval = 34.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933083 bytes = 3265808 } seq expand size = 6531581 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.5138 value = 3.3708 dict = 1.00 all = 3.5144 } Zip/UnZip{ index = 0.0951 value = 0.2967 dict = 1.00 all = 0.2845 } ---------------------------- total value len = 0.114623 GB avg = 0.123 KB (by entry num) total key len = 0.026126 GB avg = 0.028 KB total ukey len = 0.018662 GB avg = 0.020 KB total ukey num = 0.000933083 Billion total entry num = 0.000933083 Billion write speed all = 0.002477756 MB/sec (with seq num) write speed all = 0.002346347 MB/sec (without seq num) 2024/11/23-00:39:08.333063 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236752: 933083 keys, 38191318 bytes, temperature: kWarm 2024/11/23-00:39:08.333175 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293548333105, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236752, "file_size": 38191318, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038037, "largest_seqno": 3680978865, "table_properties": {"data_size": 32066672, "index_size": 1774976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26126324, "raw_average_key_size": 28, "raw_value_size": 108091494, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933083, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293543, "oldest_key_time": 0, "file_creation_time": 1732293543, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236752, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:08.407563 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236753.sst) /dev/shm/ToplingTemp/Topling-353485-yRJBA4 2024/11/23-00:39:09.526174 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236753.sst): first pass time = 1.12's, 119.992'MB/sec 2024/11/23-00:39:13.189704 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236753.sst): old prealloc_size = 147639500, real_size = 31789520 2024/11/23-00:39:13.190355 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236753.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925704 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0298 GB avg-val = 116.99 avg-zval = 32.18 2024/11/23-00:39:13.191313 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236753.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:14.521131 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236753.sst): second pass time = 3.66's, 29.567'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.33's, 26.952'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2811.614'MB/sec (index lex order gen) rebuild zvType time = 1.17's, 0.197'MB/sec write SST data time = 0.16's, 227.571'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27145.535'MB/sec zip my value time = 3.66's, unzip length = 0.108'GB zip my value throughput = 29.567'MB/sec zip pipeline throughput = 106.624'MB/sec entries = 925704 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.99 avg-zval = 32.18 usrkeys = 925704 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.99 avg-zval = 32.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925704 bytes = 3239984 } seq expand size = 6479928 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0298 GB dict = 1.08 MB all = 0.0359 GB } UnZip/Zip{ index = 10.4442 value = 3.6360 dict = 1.00 all = 3.7403 } Zip/UnZip{ index = 0.0957 value = 0.2750 dict = 1.00 all = 0.2674 } ---------------------------- total value len = 0.229401 GB avg = 0.123 KB (by entry num) total key len = 0.052046 GB avg = 0.028 KB total ukey len = 0.037176 GB avg = 0.020 KB total ukey num = 0.001858787 Billion total entry num = 0.001858787 Billion write speed all = 0.004954059 MB/sec (with seq num) write speed all = 0.004692311 MB/sec (without seq num) 2024/11/23-00:39:14.555629 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236753: 925704 keys, 35884130 bytes, temperature: kWarm 2024/11/23-00:39:14.555684 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293554555655, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236753, "file_size": 35884130, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038075, "largest_seqno": 3680977933, "table_properties": {"data_size": 29785216, "index_size": 1772672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25919712, "raw_average_key_size": 28, "raw_value_size": 108298136, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925704, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293548, "oldest_key_time": 0, "file_creation_time": 1732293548, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236753, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:14.593498 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236754.sst) /dev/shm/ToplingTemp/Topling-353485-5BK9K4 2024/11/23-00:39:15.511508 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236754.sst): first pass time = 0.92's, 146.211'MB/sec 2024/11/23-00:39:17.163229 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236754.sst): old prealloc_size = 147639500, real_size = 35299184 2024/11/23-00:39:17.163889 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236754.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929136 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0333 GB avg-val = 116.45 avg-zval = 35.83 2024/11/23-00:39:17.165138 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236754.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:18.175086 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236754.sst): second pass time = 1.65's, 65.535'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.01's, 38.942'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2728.312'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.271'MB/sec write SST data time = 0.15's, 257.384'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 24165.465'MB/sec zip my value time = 1.65's, unzip length = 0.108'GB zip my value throughput = 65.535'MB/sec zip pipeline throughput = 65.535'MB/sec entries = 929136 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.45 avg-zval = 35.83 usrkeys = 929136 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.45 avg-zval = 35.83 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929136 bytes = 3251984 } seq expand size = 6503952 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.4459 value = 3.2505 dict = 1.00 all = 3.4062 } Zip/UnZip{ index = 0.0957 value = 0.3076 dict = 1.00 all = 0.2936 } ---------------------------- total value len = 0.344107 GB avg = 0.123 KB (by entry num) total key len = 0.078062 GB avg = 0.028 KB total ukey len = 0.055758 GB avg = 0.020 KB total ukey num = 0.002787923 Billion total entry num = 0.002787923 Billion write speed all = 0.007430580 MB/sec (with seq num) write speed all = 0.007038019 MB/sec (without seq num) 2024/11/23-00:39:18.211809 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236754: 929136 keys, 39403740 bytes, temperature: kWarm 2024/11/23-00:39:18.211876 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293558211843, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236754, "file_size": 39403740, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038106, "largest_seqno": 3680978655, "table_properties": {"data_size": 33287776, "index_size": 1778944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26015808, "raw_average_key_size": 28, "raw_value_size": 108202064, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929136, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293554, "oldest_key_time": 0, "file_creation_time": 1732293554, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236754, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:18.259072 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236755.sst) /dev/shm/ToplingTemp/Topling-353485-lieU03 2024/11/23-00:39:19.124064 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236755.sst): first pass time = 0.86's, 155.174'MB/sec 2024/11/23-00:39:21.565410 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236755.sst): old prealloc_size = 147639500, real_size = 32769584 2024/11/23-00:39:21.565999 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236755.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925969 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0308 GB avg-val = 116.95 avg-zval = 33.24 2024/11/23-00:39:21.566754 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236755.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:22.862698 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236755.sst): second pass time = 2.44's, 44.370'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.30's, 28.418'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3078.832'MB/sec (index lex order gen) rebuild zvType time = 1.05's, 0.220'MB/sec write SST data time = 0.24's, 152.065'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33569.902'MB/sec zip my value time = 2.44's, unzip length = 0.108'GB zip my value throughput = 44.370'MB/sec zip pipeline throughput = 137.761'MB/sec entries = 925969 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.95 avg-zval = 33.24 usrkeys = 925969 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.95 avg-zval = 33.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925969 bytes = 3240912 } seq expand size = 6481783 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5128 value = 3.5186 dict = 1.00 all = 3.6408 } Zip/UnZip{ index = 0.0951 value = 0.2842 dict = 1.00 all = 0.2747 } ---------------------------- total value len = 0.458880 GB avg = 0.124 KB (by entry num) total key len = 0.103989 GB avg = 0.028 KB total ukey len = 0.074278 GB avg = 0.020 KB total ukey num = 0.003713892 Billion total entry num = 0.003713892 Billion write speed all = 0.009906211 MB/sec (with seq num) write speed all = 0.009383310 MB/sec (without seq num) 2024/11/23-00:39:22.903210 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236755: 925969 keys, 36865396 bytes, temperature: kWarm 2024/11/23-00:39:22.903286 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293562903243, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236755, "file_size": 36865396, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038136, "largest_seqno": 3680977583, "table_properties": {"data_size": 30776288, "index_size": 1761600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25927132, "raw_average_key_size": 28, "raw_value_size": 108290710, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925969, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293558, "oldest_key_time": 0, "file_creation_time": 1732293558, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236755, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:22.952493 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236756.sst) /dev/shm/ToplingTemp/Topling-353485-Z6rWA6 2024/11/23-00:39:23.912137 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236756.sst): first pass time = 0.96's, 139.869'MB/sec 2024/11/23-00:39:26.227524 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236756.sst): old prealloc_size = 147639500, real_size = 32097744 2024/11/23-00:39:26.228230 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236756.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922888 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0301 GB avg-val = 117.43 avg-zval = 32.63 2024/11/23-00:39:26.229028 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236756.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:27.283199 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236756.sst): second pass time = 2.31's, 46.842'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 34.278'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2561.602'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.254'MB/sec write SST data time = 0.15's, 243.880'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 30114.055'MB/sec zip my value time = 2.31's, unzip length = 0.108'GB zip my value throughput = 46.842'MB/sec zip pipeline throughput = 131.933'MB/sec entries = 922888 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.43 avg-zval = 32.63 usrkeys = 922888 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.43 avg-zval = 32.63 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922888 bytes = 3230128 } seq expand size = 6460216 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0362 GB } UnZip/Zip{ index = 10.5126 value = 3.5992 dict = 1.00 all = 3.7091 } Zip/UnZip{ index = 0.0951 value = 0.2778 dict = 1.00 all = 0.2696 } ---------------------------- total value len = 0.573717 GB avg = 0.124 KB (by entry num) total key len = 0.129830 GB avg = 0.028 KB total ukey len = 0.092736 GB avg = 0.020 KB total ukey num = 0.004636780 Billion total entry num = 0.004636780 Billion write speed all = 0.012381111 MB/sec (with seq num) write speed all = 0.011728322 MB/sec (without seq num) 2024/11/23-00:39:27.314015 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236756: 922888 keys, 36185980 bytes, temperature: kWarm 2024/11/23-00:39:27.314078 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293567314046, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236756, "file_size": 36185980, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3515979727, "largest_seqno": 3680977696, "table_properties": {"data_size": 30111040, "index_size": 1755776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25840864, "raw_average_key_size": 28, "raw_value_size": 108376892, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922888, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293562, "oldest_key_time": 0, "file_creation_time": 1732293562, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236756, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:27.353358 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236757.sst) /dev/shm/ToplingTemp/Topling-353485-gDdPb5 2024/11/23-00:39:28.231575 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236757.sst): first pass time = 0.88's, 152.840'MB/sec 2024/11/23-00:39:30.049164 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236757.sst): old prealloc_size = 147639500, real_size = 35609664 2024/11/23-00:39:30.049830 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236757.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928036 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0336 GB avg-val = 116.63 avg-zval = 36.21 2024/11/23-00:39:30.050886 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236757.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:31.195697 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236757.sst): second pass time = 1.82's, 59.599'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.15's, 34.635'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2748.555'MB/sec (index lex order gen) rebuild zvType time = 0.97's, 0.238'MB/sec write SST data time = 0.17's, 230.304'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35525.912'MB/sec zip my value time = 1.82's, unzip length = 0.108'GB zip my value throughput = 59.599'MB/sec zip pipeline throughput = 59.599'MB/sec entries = 928036 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.63 avg-zval = 36.21 usrkeys = 928036 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.07 avg-zval = 36.21 TagRS{ kinds = RS_Key0_TagN bytes = 145120 } TagArray{ size = 854676 bytes = 3098208 } seq expand size = 5982732 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0397 GB } UnZip/Zip{ index = 10.4426 value = 3.2212 dict = 1.00 all = 3.3799 } Zip/UnZip{ index = 0.0958 value = 0.3104 dict = 1.00 all = 0.2959 } ---------------------------- total value len = 0.687932 GB avg = 0.124 KB (by entry num) total key len = 0.155815 GB avg = 0.028 KB total ukey len = 0.111296 GB avg = 0.020 KB total ukey num = 0.005564816 Billion total entry num = 0.005564816 Billion write speed all = 0.014847358 MB/sec (with seq num) write speed all = 0.014063969 MB/sec (without seq num) 2024/11/23-00:39:31.231902 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236757: 928036 keys, 39710212 bytes, temperature: kWarm 2024/11/23-00:39:31.231966 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293571231934, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236757, "file_size": 39710212, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3680978954, "table_properties": {"data_size": 33600048, "index_size": 1777408, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25985008, "raw_average_key_size": 28, "raw_value_size": 108232796, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928036, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293567, "oldest_key_time": 0, "file_creation_time": 1732293567, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236757, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:31.270753 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236758.sst) /dev/shm/ToplingTemp/Topling-353485-YdI505 2024/11/23-00:39:32.205488 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236758.sst): first pass time = 0.93's, 143.597'MB/sec 2024/11/23-00:39:33.873048 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236758.sst): old prealloc_size = 147639500, real_size = 36397632 2024/11/23-00:39:33.873588 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236758.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926901 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.13 2024/11/23-00:39:33.873633 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236758.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:34.652599 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236758.sst): second pass time = 1.67's, 64.951'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 47.792'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3393.494'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.365'MB/sec write SST data time = 0.14's, 257.798'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36957.411'MB/sec zip my value time = 1.67's, unzip length = 0.108'GB zip my value throughput = 64.951'MB/sec zip pipeline throughput = 64.951'MB/sec entries = 926901 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 37.13 usrkeys = 926901 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.80 avg-zval = 37.13 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5838 value = 3.1459 dict = 1.00 all = 3.6026 } Zip/UnZip{ index = 0.0945 value = 0.3179 dict = 1.00 all = 0.2776 } ---------------------------- total value len = 0.796197 GB avg = 0.123 KB (by entry num) total key len = 0.181768 GB avg = 0.028 KB total ukey len = 0.129834 GB avg = 0.020 KB total ukey num = 0.006491717 Billion total entry num = 0.006491717 Billion write speed all = 0.017208133 MB/sec (with seq num) write speed all = 0.016294314 MB/sec (without seq num) 2024/11/23-00:39:34.688370 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236758: 926901 keys, 37255802 bytes, temperature: kWarm 2024/11/23-00:39:34.688436 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293574688404, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236758, "file_size": 37255802, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34414192, "index_size": 1751552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953228, "raw_average_key_size": 28, "raw_value_size": 108264630, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926901, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293571, "oldest_key_time": 0, "file_creation_time": 1732293571, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236758, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:34.723048 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236759.sst) /dev/shm/ToplingTemp/Topling-353485-ZJLbS5 2024/11/23-00:39:35.460649 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236759.sst): first pass time = 0.74's, 181.975'MB/sec 2024/11/23-00:39:37.090977 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236759.sst): old prealloc_size = 147639500, real_size = 36435120 2024/11/23-00:39:37.091498 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236759.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927074 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.78 avg-zval = 37.16 2024/11/23-00:39:37.091542 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236759.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:37.838417 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236759.sst): second pass time = 1.63's, 66.454'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.75's, 49.896'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3509.515'MB/sec (index lex order gen) rebuild zvType time = 0.62's, 0.373'MB/sec write SST data time = 0.13's, 296.846'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31341.184'MB/sec zip my value time = 1.63's, unzip length = 0.108'GB zip my value throughput = 66.454'MB/sec zip pipeline throughput = 66.454'MB/sec entries = 927074 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 37.16 usrkeys = 927074 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.78 avg-zval = 37.16 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5819 value = 3.1424 dict = 1.00 all = 3.5990 } Zip/UnZip{ index = 0.0945 value = 0.3182 dict = 1.00 all = 0.2779 } ---------------------------- total value len = 0.904457 GB avg = 0.122 KB (by entry num) total key len = 0.207726 GB avg = 0.028 KB total ukey len = 0.148376 GB avg = 0.020 KB total ukey num = 0.007418791 Billion total entry num = 0.007418791 Billion write speed all = 0.019568713 MB/sec (with seq num) write speed all = 0.018524452 MB/sec (without seq num) 2024/11/23-00:39:37.873378 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236759: 927074 keys, 37293416 bytes, temperature: kWarm 2024/11/23-00:39:37.873444 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293577873412, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236759, "file_size": 37293416, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34450976, "index_size": 1752192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25958072, "raw_average_key_size": 28, "raw_value_size": 108259712, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927074, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293574, "oldest_key_time": 0, "file_creation_time": 1732293574, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236759, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:37.907931 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236760.sst) /dev/shm/ToplingTemp/Topling-353485-vR3P33 2024/11/23-00:39:38.774462 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236760.sst): first pass time = 0.87's, 154.899'MB/sec 2024/11/23-00:39:40.170637 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236760.sst): old prealloc_size = 147639500, real_size = 35210720 2024/11/23-00:39:40.171214 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236760.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928127 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 116.61 avg-zval = 35.78 2024/11/23-00:39:40.171893 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236760.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:41.109880 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236760.sst): second pass time = 1.40's, 77.560'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 41.172'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3185.113'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.287'MB/sec write SST data time = 0.13's, 293.834'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36059.238'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.560'MB/sec zip pipeline throughput = 77.560'MB/sec entries = 928127 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.61 avg-zval = 35.78 usrkeys = 928127 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.72 avg-zval = 35.78 TagRS{ kinds = RS_Key0_TagN bytes = 145120 } TagArray{ size = 677691 bytes = 2456640 } seq expand size = 4743837 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5125 value = 3.2587 dict = 1.00 all = 3.4709 } Zip/UnZip{ index = 0.0951 value = 0.3069 dict = 1.00 all = 0.2881 } ---------------------------- total value len = 1.017431 GB avg = 0.122 KB (by entry num) total key len = 0.233714 GB avg = 0.028 KB total ukey len = 0.166938 GB avg = 0.020 KB total ukey num = 0.008346918 Billion total entry num = 0.008346918 Billion write speed all = 0.022012457 MB/sec (with seq num) write speed all = 0.020837621 MB/sec (without seq num) 2024/11/23-00:39:41.144074 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236760: 928127 keys, 38669770 bytes, temperature: kWarm 2024/11/23-00:39:41.144133 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293581144102, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236760, "file_size": 38669770, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3680978468, "table_properties": {"data_size": 33212752, "index_size": 1765760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25987556, "raw_average_key_size": 28, "raw_value_size": 108230182, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928127, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293577, "oldest_key_time": 0, "file_creation_time": 1732293577, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236760, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:41.182796 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236761.sst) /dev/shm/ToplingTemp/Topling-353485-yi6ZL5 2024/11/23-00:39:42.016969 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236761.sst): first pass time = 0.83's, 160.907'MB/sec 2024/11/23-00:39:43.364378 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236761.sst): old prealloc_size = 147639500, real_size = 33317920 2024/11/23-00:39:43.364973 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236761.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932573 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1081 GB zip-val = 0.0313 GB avg-val = 115.92 avg-zval = 33.56 2024/11/23-00:39:43.365764 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236761.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:44.394864 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236761.sst): second pass time = 1.35's, 80.290'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 36.439'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3139.795'MB/sec (index lex order gen) rebuild zvType time = 0.90's, 0.259'MB/sec write SST data time = 0.13's, 290.906'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 32170.925'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.290'MB/sec zip pipeline throughput = 80.290'MB/sec entries = 932573 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.92 avg-zval = 33.56 usrkeys = 932573 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.92 avg-zval = 33.56 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932573 bytes = 3380592 } seq expand size = 6528011 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4421 value = 3.4540 dict = 1.00 all = 3.5745 } Zip/UnZip{ index = 0.0958 value = 0.2895 dict = 1.00 all = 0.2798 } ---------------------------- total value len = 1.132064 GB avg = 0.122 KB (by entry num) total key len = 0.259826 GB avg = 0.028 KB total ukey len = 0.185590 GB avg = 0.020 KB total ukey num = 0.009279491 Billion total entry num = 0.009279491 Billion write speed all = 0.024487303 MB/sec (with seq num) write speed all = 0.023181282 MB/sec (without seq num) 2024/11/23-00:39:44.429481 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236761: 932573 keys, 37548974 bytes, temperature: kWarm 2024/11/23-00:39:44.429557 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293584429515, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236761, "file_size": 37548974, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701386, "largest_seqno": 3680978499, "table_properties": {"data_size": 31298448, "index_size": 1786176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26112044, "raw_average_key_size": 28, "raw_value_size": 108105726, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 932573, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293581, "oldest_key_time": 0, "file_creation_time": 1732293581, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236761, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:44.464698 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236762.sst) /dev/shm/ToplingTemp/Topling-353485-znNBF6 2024/11/23-00:39:45.392593 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236762.sst): first pass time = 0.93's, 144.654'MB/sec 2024/11/23-00:39:46.800898 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236762.sst): old prealloc_size = 147639500, real_size = 33221024 2024/11/23-00:39:46.801425 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236762.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926132 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0312 GB avg-val = 116.92 avg-zval = 33.71 2024/11/23-00:39:46.802308 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236762.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:47.732426 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236762.sst): second pass time = 1.41's, 76.930'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.185'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3503.068'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 287.029'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34232.085'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.930'MB/sec zip pipeline throughput = 76.930'MB/sec entries = 926132 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.92 avg-zval = 33.71 usrkeys = 926132 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.92 avg-zval = 33.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926132 bytes = 3357248 } seq expand size = 6482924 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.4422 value = 3.4690 dict = 1.00 all = 3.5855 } Zip/UnZip{ index = 0.0958 value = 0.2883 dict = 1.00 all = 0.2789 } ---------------------------- total value len = 1.246833 GB avg = 0.122 KB (by entry num) total key len = 0.285757 GB avg = 0.028 KB total ukey len = 0.204112 GB avg = 0.020 KB total ukey num = 0.010205623 Billion total entry num = 0.010205623 Billion write speed all = 0.026961046 MB/sec (with seq num) write speed all = 0.025524763 MB/sec (without seq num) 2024/11/23-00:39:47.765592 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236762: 926132 keys, 37433226 bytes, temperature: kWarm 2024/11/23-00:39:47.765658 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293587765623, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236762, "file_size": 37433226, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3287628183, "largest_seqno": 3680978514, "table_properties": {"data_size": 31215504, "index_size": 1773824, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25931696, "raw_average_key_size": 28, "raw_value_size": 108286088, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926132, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293584, "oldest_key_time": 0, "file_creation_time": 1732293584, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236762, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:47.830982 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236763.sst) /dev/shm/ToplingTemp/Topling-353485-FrdZE3 2024/11/23-00:39:48.660038 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236763.sst): first pass time = 0.83's, 161.900'MB/sec 2024/11/23-00:39:50.002059 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236763.sst): old prealloc_size = 147639500, real_size = 32758720 2024/11/23-00:39:50.002613 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236763.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 932032 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0308 GB avg-val = 116.01 avg-zval = 33.00 2024/11/23-00:39:50.003326 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236763.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-00:39:50.956843 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236763.sst): second pass time = 1.34's, 80.633'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.375'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3324.780'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 279.964'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33948.797'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.633'MB/sec zip pipeline throughput = 80.633'MB/sec entries = 932032 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.01 avg-zval = 33.00 usrkeys = 932032 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.01 avg-zval = 33.00 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 932032 bytes = 3029120 } seq expand size = 6524224 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5152 value = 3.5158 dict = 1.00 all = 3.6633 } Zip/UnZip{ index = 0.0951 value = 0.2844 dict = 1.00 all = 0.2730 } ---------------------------- total value len = 1.361478 GB avg = 0.122 KB (by entry num) total key len = 0.311854 GB avg = 0.028 KB total ukey len = 0.222753 GB avg = 0.020 KB total ukey num = 0.011137655 Billion total entry num = 0.011137655 Billion write speed all = 0.029435282 MB/sec (with seq num) write speed all = 0.027867919 MB/sec (without seq num) 2024/11/23-00:39:50.993258 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236763: 932032 keys, 36638798 bytes, temperature: kWarm 2024/11/23-00:39:50.993324 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293590993292, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236763, "file_size": 36638798, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3621075956, "largest_seqno": 3680978528, "table_properties": {"data_size": 30752816, "index_size": 1772736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26096896, "raw_average_key_size": 28, "raw_value_size": 108120876, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 932032, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293587, "oldest_key_time": 0, "file_creation_time": 1732293587, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236763, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:51.026877 140024718255680 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236764.sst) /dev/shm/ToplingTemp/Topling-353485-ElLto5 2024/11/23-00:39:51.127084 140024718255680 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236764.sst): first pass time = 0.10's, 239.035'MB/sec 2024/11/23-00:39:51.348115 140024718255680 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236764.sst): old prealloc_size = 147639500, real_size = 7049168 2024/11/23-00:39:51.348263 140024718255680 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236764.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 164714 prefix = 4 raw-key = 0.0026 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0193 GB zip-val = 0.0067 GB avg-val = 117.35 avg-zval = 40.64 2024/11/23-00:39:51.348438 140024718255680 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236764.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.024 GB 2024/11/23-00:39:51.500086 140024718255680 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00/236764.sst): second pass time = 0.22's, 87.684'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 51.392'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2600.782'MB/sec (index lex order gen) rebuild zvType time = 0.13's, 0.320'MB/sec write SST data time = 0.02's, 334.810'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.202'MB, throughput = 6645.530'MB/sec zip my value time = 0.22's, unzip length = 0.019'GB zip my value throughput = 87.684'MB/sec zip pipeline throughput = 87.684'MB/sec entries = 164714 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.35 avg-zval = 40.64 usrkeys = 164714 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.35 avg-zval = 40.64 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 164714 bytes = 597104 } seq expand size = 1152998 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0033 GB value = 0.0193 GB dict = 0.20 MB all = 0.0239 GB } __ZipSize{ index = 0.0003 GB value = 0.0067 GB dict = 0.20 MB all = 0.0078 GB } UnZip/Zip{ index = 10.4769 value = 2.8879 dict = 1.00 all = 3.0658 } Zip/UnZip{ index = 0.0954 value = 0.3463 dict = 1.00 all = 0.3262 } ---------------------------- total value len = 1.381961 GB avg = 0.122 KB (by entry num) total key len = 0.316466 GB avg = 0.028 KB total ukey len = 0.226047 GB avg = 0.020 KB total ukey num = 0.011302369 Billion total entry num = 0.011302369 Billion write speed all = 0.029876437 MB/sec (with seq num) write speed all = 0.028285910 MB/sec (without seq num) 2024/11/23-00:39:51.513880 140024718255680 [compaction_job.cc:1948] [default] [JOB 15433] Generated table #236764: 164714 keys, 7809325 bytes, temperature: kWarm 2024/11/23-00:39:51.513939 140024718255680 EVENT_LOG_v1 {"time_micros": 1732293591513908, "cf_name": "default", "job": 15433, "event": "table_file_creation", "file_number": 236764, "file_size": 7809325, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3679461096, "table_properties": {"data_size": 6693376, "index_size": 314432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4611992, "raw_average_key_size": 28, "raw_value_size": 19329952, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 164714, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732293591, "oldest_key_time": 0, "file_creation_time": 1732293591, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 236764, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-00:39:51.524898 140024718255680 [compaction_job.cc:703] job-15433: subcompact[0], size: 0.458890 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-15433/att-00 [236752,236753,236754,236755,236756,236757,236758,236759,236760,236761,236762,236763,236764] 2024/11/23-00:39:51.625022 140024718255680 (Original Log Time 2024/11/23-00:39:03.433552) EVENT_LOG_v1 {"time_micros": 1732293543433506, "job": 15433, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [236745], "files_L3": [236683, 236684, 236685, 236686, 236687, 236688, 236689, 236690, 236691, 236692, 236693, 236712], "score": 0.999947, "input_data_size": 483071714, "oldest_snapshot_seqno": 3680998539} 2024/11/23-00:39:51.625026 140024718255680 (Original Log Time 2024/11/23-00:39:51.624251) [compaction_job.cc:2022] [default] [JOB 15433] Compacted 1@2 + 12@3 files to L3 => 458890087 bytes 2024/11/23-00:39:51.625027 140024718255680 (Original Log Time 2024/11/23-00:39:51.624955) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 10.0 rd, 9.5 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(27.1, 433.6 +0.0 blob) out(437.6 +0.0 blob), read-write-amplify(33.1) write-amplify(16.1) OK, records in: 11369910, records dropped: 67541 output_compression: Snappy 2024/11/23-00:39:51.625034 140024718255680 (Original Log Time 2024/11/23-00:39:51.625002) EVENT_LOG_v1 {"time_micros": 1732293591624979, "cf": "default", "job": 15433, "event": "compaction_finished", "compaction_time_micros": 48091353, "compaction_time_cpu_micros": 19812072, "output_level": 3, "num_output_files": 13, "total_output_size": 458890087, "num_input_records": 11369910, "num_output_records": 11302369, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 333979145, "file_range_sync_nanos": 0, "file_fsync_nanos": 11894, "file_prepare_write_nanos": 28997, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/23-00:39:51.759571 140024718255680 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732293543,"executesMs":48327,"compactionJobId":15433,"attempt":0,"compactionInputRawBytes":1644146412,"compactionInputZipBytes":483071714,"compactionOutputRawBytes":0,"compactionOutputZipBytes":458890087,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-e397501962f9540085384d32bdbce316-104b586d4acd05e5-00"}}, response = 2024/11/23-00:39:51.759651 140024718255680 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-15433/att-00: olev 3, work 48.328 s, result 4.924 ms, install 0.803 ms, input{raw 1.531 GiB zip 460.693 MiB}