2024/11/21-23:01:26.564482 139783581591104 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00: fork to child time = 0.011308 sec 2024/11/21-23:01:26.565640 139783581591104 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00 2024/11/21-23:01:26.574202 139783581591104 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00 2024/11/21-23:01:26.576103 139783581591104 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/21-23:01:26.576481 139783581591104 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/21-23:01:26.654946 139783581591104 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T10.40.08/.rocksdb/job-28197/att-00/MANIFEST-130099 2024/11/21-23:01:26.681310 139783581591104 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T10.40.08/.rocksdb/job-28197/att-00/MANIFEST-130099 succeeded,manifest_file_number is 130099, next_file_number is 168411, last_sequence is 2691770100, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 168257 2024/11/21-23:01:26.681322 139783581591104 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/21-23:01:26.681438 139783581591104 [version_set.cc:5721] Creating manifest 168411 2024/11/21-23:01:26.726625 139783581591104 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/21-23:01:26.752025 139783581591104 [compaction_job.cc:2437] [default] [JOB 28197] Compacting 1@2 + 12@3 files to L3, score 0.82, subcompactions 1 : 1 2024/11/21-23:01:26.752049 139783581591104 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 27279 Base level 2, inputs: [168409(25MB)], [168334(37MB) 168335(37MB) 168336(37MB) 168337(38MB) 168338(36MB) 168339(35MB) 168340(35MB) 168341(35MB) 168342(35MB) 168343(36MB) 168344(37MB) 168373(37MB)] 2024/11/21-23:01:26.760448 139783581591104 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/21-23:01:26.760487 139783581591104 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/21-23:01:26.760524 139783581591104 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/21-23:01:26.760533 139783581591104 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-23:01:26.760548 139783581591104 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/21-23:01:27.253123 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168412.sst) /dev/shm/ToplingTemp/Topling-604787-JSYT9k 2024/11/21-23:01:28.018713 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168412.sst): first pass time = 0.77's, 175.322'MB/sec 2024/11/21-23:01:31.403229 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168412.sst): old prealloc_size = 147639500, real_size = 35399856 2024/11/21-23:01:31.403803 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168412.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922144 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0334 GB avg-val = 117.55 avg-zval = 36.24 2024/11/21-23:01:31.404588 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168412.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:01:32.635615 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168412.sst): second pass time = 3.38's, 32.040'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.23's, 31.854'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3158.710'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.241'MB/sec write SST data time = 0.27's, 143.343'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 32498.815'MB/sec zip my value time = 3.38's, unzip length = 0.108'GB zip my value throughput = 32.040'MB/sec zip pipeline throughput = 130.975'MB/sec entries = 922144 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.55 avg-zval = 36.24 usrkeys = 922144 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.55 avg-zval = 36.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922144 bytes = 2996976 } seq expand size = 6455008 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.5133 value = 3.2440 dict = 1.00 all = 3.4191 } Zip/UnZip{ index = 0.0951 value = 0.3083 dict = 1.00 all = 0.2925 } ---------------------------- total value len = 360.106053 GB avg = 0.342 KB (by entry num) total key len = 27.072137 GB avg = 0.026 KB total ukey len = 28.505761 GB avg = 0.015 KB total ukey num = 1.961201894 Billion total entry num = 1.054217599 Billion write speed all = 7.612837408 MB/sec (with seq num) write speed all = 7.447010151 MB/sec (without seq num) 2024/11/21-23:01:32.675492 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168412: 922144 keys, 39255590 bytes, temperature: kWarm 2024/11/21-23:01:32.675605 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201292675538, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168412, "file_size": 39255590, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2635809963, "largest_seqno": 2691725357, "table_properties": {"data_size": 33414880, "index_size": 1754240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25820032, "raw_average_key_size": 28, "raw_value_size": 108397804, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922144, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201287, "oldest_key_time": 0, "file_creation_time": 1732201287, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168412, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:01:32.769192 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168413.sst) /dev/shm/ToplingTemp/Topling-604787-AJHEEk 2024/11/21-23:01:33.789431 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168413.sst): first pass time = 1.02's, 131.559'MB/sec 2024/11/21-23:01:37.113093 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168413.sst): old prealloc_size = 147639500, real_size = 35537136 2024/11/21-23:01:37.113668 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168413.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922119 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0335 GB avg-val = 117.55 avg-zval = 36.37 2024/11/21-23:01:37.114439 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168413.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:01:38.370585 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168413.sst): second pass time = 3.32's, 32.628'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.26's, 31.327'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3168.205'MB/sec (index lex order gen) rebuild zvType time = 1.00's, 0.230'MB/sec write SST data time = 0.25's, 155.208'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 36737.360'MB/sec zip my value time = 3.32's, unzip length = 0.108'GB zip my value throughput = 32.628'MB/sec zip pipeline throughput = 135.300'MB/sec entries = 922119 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.55 avg-zval = 36.37 usrkeys = 922119 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.55 avg-zval = 36.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922119 bytes = 2996896 } seq expand size = 6454833 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0335 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.4456 value = 3.2318 dict = 1.00 all = 3.4072 } Zip/UnZip{ index = 0.0957 value = 0.3094 dict = 1.00 all = 0.2935 } ---------------------------- total value len = 360.220907 GB avg = 0.341 KB (by entry num) total key len = 27.097957 GB avg = 0.026 KB total ukey len = 28.524203 GB avg = 0.015 KB total ukey num = 1.962124013 Billion total entry num = 1.055139718 Billion write speed all = 7.614744703 MB/sec (with seq num) write speed all = 7.448791111 MB/sec (without seq num) 2024/11/21-23:01:38.408880 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168413: 922119 keys, 39392676 bytes, temperature: kWarm 2024/11/21-23:01:38.408955 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201298408913, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168413, "file_size": 39392676, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2635810008, "largest_seqno": 2691725443, "table_properties": {"data_size": 33540832, "index_size": 1765568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25819332, "raw_average_key_size": 28, "raw_value_size": 108398494, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922119, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201292, "oldest_key_time": 0, "file_creation_time": 1732201292, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168413, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:01:38.499805 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168414.sst) /dev/shm/ToplingTemp/Topling-604787-n0QBZk 2024/11/21-23:01:40.430302 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168414.sst): first pass time = 1.93's, 69.527'MB/sec 2024/11/21-23:01:42.156644 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168414.sst): old prealloc_size = 147639500, real_size = 35543872 2024/11/21-23:01:42.157169 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168414.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922117 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0336 GB avg-val = 117.55 avg-zval = 36.39 2024/11/21-23:01:42.157909 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168414.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:01:43.342058 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168414.sst): second pass time = 1.72's, 62.844'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.19's, 33.238'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3465.560'MB/sec (index lex order gen) rebuild zvType time = 0.97's, 0.238'MB/sec write SST data time = 0.22's, 181.896'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 35103.305'MB/sec zip my value time = 1.72's, unzip length = 0.108'GB zip my value throughput = 62.844'MB/sec zip pipeline throughput = 62.844'MB/sec entries = 922117 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.55 avg-zval = 36.39 usrkeys = 922117 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.55 avg-zval = 36.39 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922117 bytes = 2996896 } seq expand size = 6454819 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5142 value = 3.2301 dict = 1.00 all = 3.4066 } Zip/UnZip{ index = 0.0951 value = 0.3096 dict = 1.00 all = 0.2935 } ---------------------------- total value len = 360.335760 GB avg = 0.341 KB (by entry num) total key len = 27.123776 GB avg = 0.026 KB total ukey len = 28.542646 GB avg = 0.015 KB total ukey num = 1.963046130 Billion total entry num = 1.056061835 Billion write speed all = 7.616765885 MB/sec (with seq num) write speed all = 7.450683495 MB/sec (without seq num) 2024/11/21-23:01:43.380943 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168414: 922117 keys, 39399336 bytes, temperature: kWarm 2024/11/21-23:01:43.381109 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201303380983, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168414, "file_size": 39399336, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2635810046, "largest_seqno": 2691725320, "table_properties": {"data_size": 33559088, "index_size": 1754048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25819276, "raw_average_key_size": 28, "raw_value_size": 108398474, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922117, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201298, "oldest_key_time": 0, "file_creation_time": 1732201298, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168414, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:01:43.464720 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168415.sst) /dev/shm/ToplingTemp/Topling-604787-8Gwgrn 2024/11/21-23:01:44.836426 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168415.sst): first pass time = 1.37's, 97.851'MB/sec 2024/11/21-23:01:46.845977 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168415.sst): old prealloc_size = 147639500, real_size = 36473120 2024/11/21-23:01:46.846593 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168415.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925540 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 117.02 avg-zval = 37.24 2024/11/21-23:01:46.847265 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168415.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:01:47.705829 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168415.sst): second pass time = 2.01's, 53.914'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 46.411'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2987.118'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.333'MB/sec write SST data time = 0.16's, 243.573'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31156.344'MB/sec zip my value time = 2.01's, unzip length = 0.108'GB zip my value throughput = 53.914'MB/sec zip pipeline throughput = 67.798'MB/sec entries = 925540 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.02 avg-zval = 37.24 usrkeys = 925540 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.27 avg-zval = 37.24 TagRS{ kinds = RS_Key0_TagN bytes = 144720 } TagArray{ size = 694070 bytes = 2429264 } seq expand size = 4858490 multi value expand size = 0 cnt WriteAppend = 19 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0399 GB } UnZip/Zip{ index = 10.4431 value = 3.1420 dict = 1.00 all = 3.3634 } Zip/UnZip{ index = 0.0958 value = 0.3183 dict = 1.00 all = 0.2973 } ---------------------------- total value len = 360.448921 GB avg = 0.341 KB (by entry num) total key len = 27.149691 GB avg = 0.026 KB total ukey len = 28.561157 GB avg = 0.015 KB total ukey num = 1.963971670 Billion total entry num = 1.056987375 Billion write speed all = 7.618846300 MB/sec (with seq num) write speed all = 7.452632612 MB/sec (without seq num) 2024/11/21-23:01:47.739792 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168415: 925540 keys, 39905798 bytes, temperature: kWarm 2024/11/21-23:01:47.739867 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201307739821, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168415, "file_size": 39905798, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 2691725498, "table_properties": {"data_size": 34469008, "index_size": 1772544, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25915120, "raw_average_key_size": 28, "raw_value_size": 108302704, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925540, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201303, "oldest_key_time": 0, "file_creation_time": 1732201303, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168415, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:01:47.784405 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168416.sst) /dev/shm/ToplingTemp/Topling-604787-4aOhUn 2024/11/21-23:01:48.813250 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168416.sst): first pass time = 1.03's, 130.459'MB/sec 2024/11/21-23:01:50.967368 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168416.sst): old prealloc_size = 147639500, real_size = 36303056 2024/11/21-23:01:50.967931 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168416.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926452 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0343 GB avg-val = 116.87 avg-zval = 37.02 2024/11/21-23:01:50.968395 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168416.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:01:51.684387 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168416.sst): second pass time = 2.15's, 50.281'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.72's, 54.273'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3268.523'MB/sec (index lex order gen) rebuild zvType time = 0.54's, 0.430'MB/sec write SST data time = 0.18's, 218.965'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 22291.279'MB/sec zip my value time = 2.15's, unzip length = 0.108'GB zip my value throughput = 50.281'MB/sec zip pipeline throughput = 107.279'MB/sec entries = 926452 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.87 avg-zval = 37.02 usrkeys = 926452 avg-key = 20.00 avg-zkey = 1.92 avg-val = 121.09 avg-zval = 37.02 TagRS{ kinds = RS_Key0_TagN bytes = 144840 } TagArray{ size = 558709 bytes = 1606304 } seq expand size = 3910963 multi value expand size = 0 cnt WriteAppend = 20 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0343 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.4413 value = 3.1571 dict = 1.00 all = 3.4492 } Zip/UnZip{ index = 0.0958 value = 0.3167 dict = 1.00 all = 0.2899 } ---------------------------- total value len = 360.561109 GB avg = 0.341 KB (by entry num) total key len = 27.175632 GB avg = 0.026 KB total ukey len = 28.579686 GB avg = 0.015 KB total ukey num = 1.964898122 Billion total entry num = 1.057913827 Billion write speed all = 7.620965441 MB/sec (with seq num) write speed all = 7.454619076 MB/sec (without seq num) 2024/11/21-23:01:51.721002 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168416: 926452 keys, 38912515 bytes, temperature: kWarm 2024/11/21-23:01:51.721084 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201311721040, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168416, "file_size": 38912515, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 2546596469, "table_properties": {"data_size": 34296640, "index_size": 1774592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25940656, "raw_average_key_size": 28, "raw_value_size": 108277176, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926452, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201307, "oldest_key_time": 0, "file_creation_time": 1732201307, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168416, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:01:51.787715 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168417.sst) /dev/shm/ToplingTemp/Topling-604787-2UTczn 2024/11/21-23:01:52.635508 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168417.sst): first pass time = 0.85's, 158.323'MB/sec 2024/11/21-23:01:54.346157 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168417.sst): old prealloc_size = 147639500, real_size = 36544560 2024/11/21-23:01:54.346727 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168417.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927140 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.77 avg-zval = 37.25 2024/11/21-23:01:54.346770 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168417.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:01:55.196277 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168417.sst): second pass time = 1.71's, 63.331'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.999'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3212.817'MB/sec (index lex order gen) rebuild zvType time = 0.62's, 0.375'MB/sec write SST data time = 0.23's, 162.149'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 36695.507'MB/sec zip my value time = 1.71's, unzip length = 0.108'GB zip my value throughput = 63.331'MB/sec zip pipeline throughput = 63.331'MB/sec entries = 927140 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.77 avg-zval = 37.25 usrkeys = 927140 avg-key = 20.00 avg-zkey = 1.92 avg-val = 116.77 avg-zval = 37.25 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.4415 value = 3.1346 dict = 1.00 all = 3.5884 } Zip/UnZip{ index = 0.0958 value = 0.3190 dict = 1.00 all = 0.2787 } ---------------------------- total value len = 360.669367 GB avg = 0.341 KB (by entry num) total key len = 27.201592 GB avg = 0.026 KB total ukey len = 28.598228 GB avg = 0.015 KB total ukey num = 1.965825262 Billion total entry num = 1.058840967 Billion write speed all = 7.623077301 MB/sec (with seq num) write speed all = 7.456596643 MB/sec (without seq num) 2024/11/21-23:01:55.231506 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168417: 927140 keys, 37402952 bytes, temperature: kWarm 2024/11/21-23:01:55.231573 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201315231543, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168417, "file_size": 37402952, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34536736, "index_size": 1775872, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25959920, "raw_average_key_size": 28, "raw_value_size": 108257896, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927140, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201311, "oldest_key_time": 0, "file_creation_time": 1732201311, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168417, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:01:55.285545 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168418.sst) /dev/shm/ToplingTemp/Topling-604787-qmL9Fn 2024/11/21-23:01:56.110711 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168418.sst): first pass time = 0.83's, 162.662'MB/sec 2024/11/21-23:01:57.632438 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168418.sst): old prealloc_size = 147639500, real_size = 36428704 2024/11/21-23:01:57.633000 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168418.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927015 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.78 avg-zval = 37.14 2024/11/21-23:01:57.633048 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168418.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:01:58.276654 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168418.sst): second pass time = 1.52's, 71.175'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.64's, 57.881'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3270.464'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.448'MB/sec write SST data time = 0.13's, 295.461'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31876.933'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.175'MB/sec zip pipeline throughput = 71.175'MB/sec entries = 927015 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.78 avg-zval = 37.14 usrkeys = 927015 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.78 avg-zval = 37.14 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5148 value = 3.1441 dict = 1.00 all = 3.5996 } Zip/UnZip{ index = 0.0951 value = 0.3181 dict = 1.00 all = 0.2778 } ---------------------------- total value len = 360.777629 GB avg = 0.340 KB (by entry num) total key len = 27.227548 GB avg = 0.026 KB total ukey len = 28.616769 GB avg = 0.015 KB total ukey num = 1.966752277 Billion total entry num = 1.059767982 Billion write speed all = 7.625253530 MB/sec (with seq num) write speed all = 7.458637205 MB/sec (without seq num) 2024/11/21-23:01:58.309096 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168418: 927015 keys, 37286584 bytes, temperature: kWarm 2024/11/21-23:01:58.309159 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201318309124, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168418, "file_size": 37286584, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34433488, "index_size": 1763264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25956420, "raw_average_key_size": 28, "raw_value_size": 108261378, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927015, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201315, "oldest_key_time": 0, "file_creation_time": 1732201315, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168418, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:01:58.347639 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168419.sst) /dev/shm/ToplingTemp/Topling-604787-5CTDXn 2024/11/21-23:01:59.190445 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168419.sst): first pass time = 0.84's, 159.257'MB/sec 2024/11/21-23:02:01.110401 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168419.sst): old prealloc_size = 147639500, real_size = 36427024 2024/11/21-23:02:01.111037 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168419.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926999 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.79 avg-zval = 37.14 2024/11/21-23:02:01.112106 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168419.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:02:01.759767 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168419.sst): second pass time = 1.92's, 56.411'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.65's, 57.419'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2870.355'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.444'MB/sec write SST data time = 0.13's, 295.166'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32490.106'MB/sec zip my value time = 1.92's, unzip length = 0.108'GB zip my value throughput = 56.411'MB/sec zip pipeline throughput = 56.411'MB/sec entries = 926999 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.14 usrkeys = 926999 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.14 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 10 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5104 value = 3.1443 dict = 1.00 all = 3.5998 } Zip/UnZip{ index = 0.0951 value = 0.3180 dict = 1.00 all = 0.2778 } ---------------------------- total value len = 360.885890 GB avg = 0.340 KB (by entry num) total key len = 27.253504 GB avg = 0.026 KB total ukey len = 28.635309 GB avg = 0.015 KB total ukey num = 1.967679276 Billion total entry num = 1.060694981 Billion write speed all = 7.627369132 MB/sec (with seq num) write speed all = 7.460618479 MB/sec (without seq num) 2024/11/21-23:02:01.801660 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168419: 926999 keys, 37285228 bytes, temperature: kWarm 2024/11/21-23:02:01.801729 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201321801690, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168419, "file_size": 37285228, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34431104, "index_size": 1763968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25955972, "raw_average_key_size": 28, "raw_value_size": 108261838, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926999, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201318, "oldest_key_time": 0, "file_creation_time": 1732201318, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168419, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:02:01.845313 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168420.sst) /dev/shm/ToplingTemp/Topling-604787-PQVfEm 2024/11/21-23:02:02.574224 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168420.sst): first pass time = 0.73's, 184.144'MB/sec 2024/11/21-23:02:04.099103 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168420.sst): old prealloc_size = 147639500, real_size = 36469600 2024/11/21-23:02:04.099644 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168420.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927202 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.76 avg-zval = 37.18 2024/11/21-23:02:04.099687 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168420.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:02:04.743164 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168420.sst): second pass time = 1.52's, 71.024'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.64's, 57.960'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3403.947'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.448'MB/sec write SST data time = 0.13's, 296.204'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31088.911'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.024'MB/sec zip pipeline throughput = 71.024'MB/sec entries = 927202 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 37.18 usrkeys = 927202 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.76 avg-zval = 37.18 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5134 value = 3.1402 dict = 1.00 all = 3.5957 } Zip/UnZip{ index = 0.0951 value = 0.3184 dict = 1.00 all = 0.2781 } ---------------------------- total value len = 360.994147 GB avg = 0.340 KB (by entry num) total key len = 27.279466 GB avg = 0.026 KB total ukey len = 28.653853 GB avg = 0.015 KB total ukey num = 1.968606478 Billion total entry num = 1.061622183 Billion write speed all = 7.629559364 MB/sec (with seq num) write speed all = 7.462672731 MB/sec (without seq num) 2024/11/21-23:02:04.779036 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168420: 927202 keys, 37327698 bytes, temperature: kWarm 2024/11/21-23:02:04.779119 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201324779076, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168420, "file_size": 37327698, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34473744, "index_size": 1763840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25961656, "raw_average_key_size": 28, "raw_value_size": 108256108, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927202, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201321, "oldest_key_time": 0, "file_creation_time": 1732201321, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168420, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:02:04.847371 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168421.sst) /dev/shm/ToplingTemp/Topling-604787-mPet1m 2024/11/21-23:02:05.798276 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168421.sst): first pass time = 0.95's, 141.155'MB/sec 2024/11/21-23:02:07.482155 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168421.sst): old prealloc_size = 147639500, real_size = 36958336 2024/11/21-23:02:07.482672 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168421.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925974 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0350 GB avg-val = 116.95 avg-zval = 37.76 2024/11/21-23:02:07.482909 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168421.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:02:08.300436 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168421.sst): second pass time = 1.68's, 64.336'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 46.936'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3551.971'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.394'MB/sec write SST data time = 0.23's, 166.806'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 18697.005'MB/sec zip my value time = 1.68's, unzip length = 0.108'GB zip my value throughput = 64.336'MB/sec zip pipeline throughput = 67.689'MB/sec entries = 925974 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.95 avg-zval = 37.76 usrkeys = 925974 avg-key = 20.00 avg-zkey = 1.90 avg-val = 118.02 avg-zval = 37.76 TagRS{ kinds = RS_Key0_TagN bytes = 144800 } TagArray{ size = 141801 bytes = 443136 } seq expand size = 992607 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0350 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5129 value = 3.0971 dict = 1.00 all = 3.4948 } Zip/UnZip{ index = 0.0951 value = 0.3229 dict = 1.00 all = 0.2861 } ---------------------------- total value len = 361.103430 GB avg = 0.340 KB (by entry num) total key len = 27.305393 GB avg = 0.026 KB total ukey len = 28.672372 GB avg = 0.015 KB total ukey num = 1.969532452 Billion total entry num = 1.062548157 Billion write speed all = 7.631682786 MB/sec (with seq num) write speed all = 7.464662265 MB/sec (without seq num) 2024/11/21-23:02:08.338489 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168421: 925974 keys, 38404859 bytes, temperature: kWarm 2024/11/21-23:02:08.338580 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201328338537, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168421, "file_size": 38404859, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 2662697074, "table_properties": {"data_size": 34965040, "index_size": 1761600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25927272, "raw_average_key_size": 28, "raw_value_size": 108290536, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925974, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201324, "oldest_key_time": 0, "file_creation_time": 1732201324, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168421, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:02:08.409505 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168422.sst) /dev/shm/ToplingTemp/Topling-604787-UaV1Kl 2024/11/21-23:02:09.301426 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168422.sst): first pass time = 0.89's, 150.491'MB/sec 2024/11/21-23:02:10.784478 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168422.sst): old prealloc_size = 147639500, real_size = 35697216 2024/11/21-23:02:10.785027 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168422.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922576 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0337 GB avg-val = 117.48 avg-zval = 36.54 2024/11/21-23:02:10.785866 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168422.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:02:11.820590 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168422.sst): second pass time = 1.48's, 73.116'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 38.176'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3326.540'MB/sec (index lex order gen) rebuild zvType time = 0.90's, 0.256'MB/sec write SST data time = 0.13's, 293.093'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 35152.762'MB/sec zip my value time = 1.48's, unzip length = 0.108'GB zip my value throughput = 73.116'MB/sec zip pipeline throughput = 73.116'MB/sec entries = 922576 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.48 avg-zval = 36.54 usrkeys = 922576 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.48 avg-zval = 36.54 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922576 bytes = 2998384 } seq expand size = 6458032 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0396 GB } UnZip/Zip{ index = 10.5098 value = 3.2152 dict = 1.00 all = 3.3933 } Zip/UnZip{ index = 0.0951 value = 0.3110 dict = 1.00 all = 0.2947 } ---------------------------- total value len = 361.218273 GB avg = 0.340 KB (by entry num) total key len = 27.331225 GB avg = 0.026 KB total ukey len = 28.690824 GB avg = 0.015 KB total ukey num = 1.970455028 Billion total entry num = 1.063470733 Billion write speed all = 7.633918855 MB/sec (with seq num) write speed all = 7.466764877 MB/sec (without seq num) 2024/11/21-23:02:11.871039 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168422: 922576 keys, 39553716 bytes, temperature: kWarm 2024/11/21-23:02:11.871101 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201331871068, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168422, "file_size": 39553716, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2635811912, "largest_seqno": 2691725734, "table_properties": {"data_size": 33710768, "index_size": 1755648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25832128, "raw_average_key_size": 28, "raw_value_size": 108385616, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922576, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201328, "oldest_key_time": 0, "file_creation_time": 1732201328, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168422, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:02:11.909105 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168423.sst) /dev/shm/ToplingTemp/Topling-604787-l0JOwm 2024/11/21-23:02:12.792465 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168423.sst): first pass time = 0.88's, 151.945'MB/sec 2024/11/21-23:02:14.431700 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168423.sst): old prealloc_size = 147639500, real_size = 35301232 2024/11/21-23:02:14.432374 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168423.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922368 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0333 GB avg-val = 117.51 avg-zval = 36.11 2024/11/21-23:02:14.433173 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168423.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/21-23:02:15.508659 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168423.sst): second pass time = 1.64's, 66.155'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.08's, 36.360'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2718.955'MB/sec (index lex order gen) rebuild zvType time = 0.86's, 0.267'MB/sec write SST data time = 0.21's, 183.398'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 31253.663'MB/sec zip my value time = 1.64's, unzip length = 0.108'GB zip my value throughput = 66.155'MB/sec zip pipeline throughput = 66.155'MB/sec entries = 922368 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.51 avg-zval = 36.11 usrkeys = 922368 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.51 avg-zval = 36.11 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922368 bytes = 2997712 } seq expand size = 6456576 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.4423 value = 3.2546 dict = 1.00 all = 3.4276 } Zip/UnZip{ index = 0.0958 value = 0.3073 dict = 1.00 all = 0.2917 } ---------------------------- total value len = 361.333121 GB avg = 0.339 KB (by entry num) total key len = 27.357051 GB avg = 0.026 KB total ukey len = 28.709271 GB avg = 0.015 KB total ukey num = 1.971377396 Billion total entry num = 1.064393101 Billion write speed all = 7.636129400 MB/sec (with seq num) write speed all = 7.468842568 MB/sec (without seq num) 2024/11/21-23:02:15.544014 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168423: 922368 keys, 39157630 bytes, temperature: kWarm 2024/11/21-23:02:15.544079 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201335544047, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168423, "file_size": 39157630, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2635811955, "largest_seqno": 2691725748, "table_properties": {"data_size": 33303840, "index_size": 1766592, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25826304, "raw_average_key_size": 28, "raw_value_size": 108391484, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922368, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201331, "oldest_key_time": 0, "file_creation_time": 1732201331, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168423, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:02:15.583294 139783581591104 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168424.sst) /dev/shm/ToplingTemp/Topling-604787-OzId9l 2024/11/21-23:02:15.653986 139783581591104 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168424.sst): first pass time = 0.07's, 235.008'MB/sec 2024/11/21-23:02:15.798812 139783581591104 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168424.sst): old prealloc_size = 147639500, real_size = 5049232 2024/11/21-23:02:15.798931 139783581591104 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168424.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 113737 prefix = 4 raw-key = 0.0018 GB zip-key = 0.0002 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0134 GB zip-val = 0.0048 GB avg-val = 118.00 avg-zval = 42.22 2024/11/21-23:02:15.799051 139783581591104 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168424.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0002 GB, waited 0.000 sec, Key+Value = 0.017 GB 2024/11/21-23:02:15.951964 139783581591104 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00/168424.sst): second pass time = 0.14's, 92.988'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 36.034'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2288.698'MB/sec (index lex order gen) rebuild zvType time = 0.10's, 0.288'MB/sec write SST data time = 0.05's, 101.592'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.140'MB, throughput = 5516.762'MB/sec zip my value time = 0.14's, unzip length = 0.013'GB zip my value throughput = 92.988'MB/sec zip pipeline throughput = 92.988'MB/sec entries = 113737 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 42.22 usrkeys = 113737 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 42.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 113737 bytes = 355440 } seq expand size = 796159 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0023 GB value = 0.0134 GB dict = 0.14 MB all = 0.0166 GB } __ZipSize{ index = 0.0002 GB value = 0.0048 GB dict = 0.14 MB all = 0.0055 GB } UnZip/Zip{ index = 10.3805 value = 2.7952 dict = 1.00 all = 3.0097 } Zip/UnZip{ index = 0.0963 value = 0.3578 dict = 1.00 all = 0.3323 } ---------------------------- total value len = 361.347339 GB avg = 0.339 KB (by entry num) total key len = 27.360236 GB avg = 0.026 KB total ukey len = 28.711546 GB avg = 0.015 KB total ukey num = 1.971491133 Billion total entry num = 1.064506838 Billion write speed all = 7.636404768 MB/sec (with seq num) write speed all = 7.469101517 MB/sec (without seq num) 2024/11/21-23:02:15.968027 139783581591104 [compaction_job.cc:1948] [default] [JOB 28197] Generated table #168424: 113737 keys, 5517343 bytes, temperature: kWarm 2024/11/21-23:02:15.968102 139783581591104 EVENT_LOG_v1 {"time_micros": 1732201335968061, "cf_name": "default", "job": 28197, "event": "table_file_creation", "file_number": 168424, "file_size": 5517343, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 2636020205, "largest_seqno": 2657646800, "table_properties": {"data_size": 4801472, "index_size": 219136, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3184636, "raw_average_key_size": 28, "raw_value_size": 13420966, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 113737, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732201335, "oldest_key_time": 0, "file_creation_time": 1732201335, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "DRFHGKF2T4P96U3FZ4Y6", "orig_file_number": 168424, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/21-23:02:15.983713 139783581591104 [compaction_job.cc:703] job-28197: subcompact[0], size: 0.468802 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-28197/att-00 [168412,168413,168414,168415,168416,168417,168418,168419,168420,168421,168422,168423,168424] 2024/11/21-23:02:16.154378 139783581591104 (Original Log Time 2024/11/21-23:01:26.752128) EVENT_LOG_v1 {"time_micros": 1732201286752069, "job": 28197, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [168409], "files_L3": [168334, 168335, 168336, 168337, 168338, 168339, 168340, 168341, 168342, 168343, 168344, 168373], "score": 0.816029, "input_data_size": 490987366, "oldest_snapshot_seqno": 2691770100} 2024/11/21-23:02:16.154383 139783581591104 (Original Log Time 2024/11/21-23:02:16.153292) [compaction_job.cc:2022] [default] [JOB 28197] Compacted 1@2 + 12@3 files to L3 => 468801925 bytes 2024/11/21-23:02:16.154385 139783581591104 (Original Log Time 2024/11/21-23:02:16.154297) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 10.0 rd, 9.5 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(25.3, 442.9 +0.0 blob) out(447.1 +0.0 blob), read-write-amplify(36.1) write-amplify(17.6) OK, records in: 11315113, records dropped: 103730 output_compression: Snappy 2024/11/21-23:02:16.154394 139783581591104 (Original Log Time 2024/11/21-23:02:16.154354) EVENT_LOG_v1 {"time_micros": 1732201336154327, "cf": "default", "job": 28197, "event": "compaction_finished", "compaction_time_micros": 49231596, "compaction_time_cpu_micros": 18748086, "output_level": 3, "num_output_files": 13, "total_output_size": 468801925, "num_input_records": 11315113, "num_output_records": 11211383, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 328564564, "file_range_sync_nanos": 0, "file_fsync_nanos": 12985, "file_prepare_write_nanos": 45674, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/21-23:02:16.287339 139783581591104 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732156808,"starts":1732201286,"executesMs":49581,"compactionJobId":28197,"attempt":0,"compactionInputRawBytes":1641948858,"compactionInputZipBytes":490987366,"compactionOutputRawBytes":0,"compactionOutputZipBytes":468801925,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-700d90289a1bc5ad0e6a756beddf3729-5de8c6b3249c93ec-00"}}, response = 2024/11/21-23:02:16.287416 139783581591104 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-21T10.40.08/.rocksdb/job-28197/att-00: olev 3, work 49.582 s, result 6.615 ms, install 1.127 ms, input{raw 1.529 GiB zip 468.242 MiB}