2024/11/23-12:26:36.168325 139957240342080 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00: fork to child time = 0.036231 sec 2024/11/23-12:26:36.169143 139957240342080 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00 2024/11/23-12:26:36.177737 139957240342080 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00 2024/11/23-12:26:36.178950 139957240342080 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-12:26:36.179296 139957240342080 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-12:26:36.230066 139957240342080 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-25975/att-00/MANIFEST-206347 2024/11/23-12:26:36.249228 139957240342080 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-25975/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 259429, last_sequence is 3973744032, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 259269 2024/11/23-12:26:36.249238 139957240342080 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-12:26:36.249365 139957240342080 [version_set.cc:5721] Creating manifest 259429 2024/11/23-12:26:36.286018 139957240342080 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-12:26:36.305327 139957240342080 [compaction_job.cc:2437] [default] [JOB 25975] Compacting 1@2 + 11@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/23-12:26:36.305342 139957240342080 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 19471 Base level 2, inputs: [259413(27MB)], [259356(35MB) 259357(35MB) 259358(35MB) 259359(34MB) 259360(35MB) 259361(37MB) 259362(35MB) 259363(34MB) 259364(33MB) 259365(34MB) 259367(35MB)] 2024/11/23-12:26:36.312704 139957240342080 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-12:26:36.312733 139957240342080 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-12:26:36.312750 139957240342080 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-12:26:36.312756 139957240342080 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-12:26:36.312768 139957240342080 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-12:26:36.949316 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259430.sst) /dev/shm/ToplingTemp/Topling-48109-ecDrKZ 2024/11/23-12:26:37.829221 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259430.sst): first pass time = 0.88's, 152.545'MB/sec 2024/11/23-12:26:41.384955 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259430.sst): old prealloc_size = 147639500, real_size = 32711568 2024/11/23-12:26:41.385556 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259430.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931700 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0307 GB avg-val = 116.06 avg-zval = 32.96 2024/11/23-12:26:41.386382 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259430.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:26:42.590835 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259430.sst): second pass time = 3.54's, 30.525'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.21's, 30.441'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3136.481'MB/sec (index lex order gen) rebuild zvType time = 0.97's, 0.239'MB/sec write SST data time = 0.23's, 159.365'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 30615.507'MB/sec zip my value time = 3.54's, unzip length = 0.108'GB zip my value throughput = 30.525'MB/sec zip pipeline throughput = 136.954'MB/sec entries = 931700 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.06 avg-zval = 32.96 usrkeys = 931700 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.06 avg-zval = 32.96 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931700 bytes = 3144496 } seq expand size = 6521900 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5114 value = 3.5215 dict = 1.00 all = 3.6564 } Zip/UnZip{ index = 0.0951 value = 0.2840 dict = 1.00 all = 0.2735 } ---------------------------- total value len = 8.963272 GB avg = 0.168 KB (by entry num) total key len = 1.466909 GB avg = 0.028 KB total ukey len = 1.016512 GB avg = 0.015 KB total ukey num = 0.066482135 Billion total entry num = 0.053289864 Billion write speed all = 0.821896885 MB/sec (with seq num) write speed all = 0.788303010 MB/sec (without seq num) 2024/11/23-12:26:42.681592 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259430: 931700 keys, 36707314 bytes, temperature: kWarm 2024/11/23-12:26:42.681713 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336002681639, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259430, "file_size": 36707314, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3901411057, "largest_seqno": 3973690247, "table_properties": {"data_size": 30705728, "index_size": 1772736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26087600, "raw_average_key_size": 28, "raw_value_size": 108130268, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931700, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732335996, "oldest_key_time": 0, "file_creation_time": 1732335996, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259430, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:26:42.766434 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259431.sst) /dev/shm/ToplingTemp/Topling-48109-5JlJqX 2024/11/23-12:26:44.545235 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259431.sst): first pass time = 1.78's, 75.456'MB/sec 2024/11/23-12:26:46.773950 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259431.sst): old prealloc_size = 147639500, real_size = 32369440 2024/11/23-12:26:46.774487 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259431.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934716 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1080 GB zip-val = 0.0304 GB avg-val = 115.59 avg-zval = 32.48 2024/11/23-12:26:46.775258 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259431.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:26:48.101450 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259431.sst): second pass time = 2.23's, 48.502'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.33's, 27.489'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3565.058'MB/sec (index lex order gen) rebuild zvType time = 1.19's, 0.196'MB/sec write SST data time = 0.13's, 273.494'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 30477.674'MB/sec zip my value time = 2.23's, unzip length = 0.108'GB zip my value throughput = 48.502'MB/sec zip pipeline throughput = 76.506'MB/sec entries = 934716 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.59 avg-zval = 32.48 usrkeys = 934716 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.59 avg-zval = 32.48 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934716 bytes = 3271520 } seq expand size = 6543012 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.5136 value = 3.5591 dict = 1.00 all = 3.6781 } Zip/UnZip{ index = 0.0951 value = 0.2810 dict = 1.00 all = 0.2719 } ---------------------------- total value len = 9.077861 GB avg = 0.167 KB (by entry num) total key len = 1.493081 GB avg = 0.028 KB total ukey len = 1.035206 GB avg = 0.015 KB total ukey num = 0.067416851 Billion total entry num = 0.054224580 Billion write speed all = 0.832627256 MB/sec (with seq num) write speed all = 0.798458974 MB/sec (without seq num) 2024/11/23-12:26:48.146637 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259431: 934716 keys, 36490918 bytes, temperature: kWarm 2024/11/23-12:26:48.146706 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336008146670, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259431, "file_size": 36490918, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3711387651, "largest_seqno": 3973690163, "table_properties": {"data_size": 30357456, "index_size": 1778112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26172048, "raw_average_key_size": 28, "raw_value_size": 108045684, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934716, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336002, "oldest_key_time": 0, "file_creation_time": 1732336002, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259431, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:26:48.185150 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259432.sst) /dev/shm/ToplingTemp/Topling-48109-B6T0TW 2024/11/23-12:26:49.252211 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259432.sst): first pass time = 1.07's, 125.788'MB/sec 2024/11/23-12:26:51.635641 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259432.sst): old prealloc_size = 147639500, real_size = 32786160 2024/11/23-12:26:51.636324 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259432.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931400 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1081 GB zip-val = 0.0308 GB avg-val = 116.10 avg-zval = 33.04 2024/11/23-12:26:51.637166 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259432.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:26:52.792330 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259432.sst): second pass time = 2.38's, 45.387'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.16's, 31.799'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2687.694'MB/sec (index lex order gen) rebuild zvType time = 1.00's, 0.232'MB/sec write SST data time = 0.15's, 241.330'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 32804.404'MB/sec zip my value time = 2.38's, unzip length = 0.108'GB zip my value throughput = 45.387'MB/sec zip pipeline throughput = 71.131'MB/sec entries = 931400 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.10 avg-zval = 33.04 usrkeys = 931400 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.10 avg-zval = 33.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931400 bytes = 3143488 } seq expand size = 6519800 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.4436 value = 3.5145 dict = 1.00 all = 3.6491 } Zip/UnZip{ index = 0.0958 value = 0.2845 dict = 1.00 all = 0.2740 } ---------------------------- total value len = 9.192519 GB avg = 0.167 KB (by entry num) total key len = 1.519160 GB avg = 0.028 KB total ukey len = 1.053834 GB avg = 0.015 KB total ukey num = 0.068348251 Billion total entry num = 0.055155980 Billion write speed all = 0.843400924 MB/sec (with seq num) write speed all = 0.808658580 MB/sec (without seq num) 2024/11/23-12:26:52.837228 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259432: 931400 keys, 36781312 bytes, temperature: kWarm 2024/11/23-12:26:52.837283 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336012837253, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259432, "file_size": 36781312, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3867815840, "largest_seqno": 3973691037, "table_properties": {"data_size": 30769440, "index_size": 1783680, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26079200, "raw_average_key_size": 28, "raw_value_size": 108138568, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931400, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336008, "oldest_key_time": 0, "file_creation_time": 1732336008, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259432, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:26:52.877033 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259433.sst) /dev/shm/ToplingTemp/Topling-48109-HZzyNW 2024/11/23-12:26:54.158611 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259433.sst): first pass time = 1.28's, 104.732'MB/sec 2024/11/23-12:26:55.585573 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259433.sst): old prealloc_size = 147639500, real_size = 31506112 2024/11/23-12:26:55.586138 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259433.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930514 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0295 GB avg-val = 116.24 avg-zval = 31.71 2024/11/23-12:26:55.586916 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259433.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:26:56.436077 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259433.sst): second pass time = 1.43's, 75.840'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 41.739'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3282.928'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 271.262'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 38846.511'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.840'MB/sec zip pipeline throughput = 75.840'MB/sec entries = 930514 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.24 avg-zval = 31.71 usrkeys = 930514 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.24 avg-zval = 31.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930514 bytes = 3140496 } seq expand size = 6513598 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.08 MB all = 0.0355 GB } UnZip/Zip{ index = 10.5144 value = 3.6661 dict = 1.00 all = 3.7810 } Zip/UnZip{ index = 0.0951 value = 0.2728 dict = 1.00 all = 0.2645 } ---------------------------- total value len = 9.307196 GB avg = 0.166 KB (by entry num) total key len = 1.545214 GB avg = 0.028 KB total ukey len = 1.072445 GB avg = 0.015 KB total ukey num = 0.069278765 Billion total entry num = 0.056086494 Billion write speed all = 0.854236553 MB/sec (with seq num) write speed all = 0.818918217 MB/sec (without seq num) 2024/11/23-12:26:56.479706 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259433: 930514 keys, 35498274 bytes, temperature: kWarm 2024/11/23-12:26:56.479773 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336016479740, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259433, "file_size": 35498274, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3845180379, "largest_seqno": 3973690740, "table_properties": {"data_size": 29503344, "index_size": 1769984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26054392, "raw_average_key_size": 28, "raw_value_size": 108163376, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930514, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336012, "oldest_key_time": 0, "file_creation_time": 1732336012, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259433, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:26:56.516525 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259434.sst) /dev/shm/ToplingTemp/Topling-48109-NyXJDY 2024/11/23-12:26:57.421390 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259434.sst): first pass time = 0.90's, 148.335'MB/sec 2024/11/23-12:26:58.855534 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259434.sst): old prealloc_size = 147639500, real_size = 32436688 2024/11/23-12:26:58.856097 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259434.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930291 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0304 GB avg-val = 116.28 avg-zval = 32.71 2024/11/23-12:26:58.856878 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259434.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:26:59.715572 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259434.sst): second pass time = 1.43's, 75.478'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 42.358'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3262.912'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 287.036'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 36146.830'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.478'MB/sec zip pipeline throughput = 75.478'MB/sec entries = 930291 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.28 avg-zval = 32.71 usrkeys = 930291 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.28 avg-zval = 32.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930291 bytes = 3139744 } seq expand size = 6512037 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.5115 value = 3.5543 dict = 1.00 all = 3.6844 } Zip/UnZip{ index = 0.0951 value = 0.2814 dict = 1.00 all = 0.2714 } ---------------------------- total value len = 9.421878 GB avg = 0.165 KB (by entry num) total key len = 1.571262 GB avg = 0.028 KB total ukey len = 1.091050 GB avg = 0.016 KB total ukey num = 0.070209056 Billion total entry num = 0.057016785 Billion write speed all = 0.865090646 MB/sec (with seq num) write speed all = 0.829195761 MB/sec (without seq num) 2024/11/23-12:26:59.760696 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259434: 930291 keys, 36428414 bytes, temperature: kWarm 2024/11/23-12:26:59.760764 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336019760730, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259434, "file_size": 36428414, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3883658141, "largest_seqno": 3973690788, "table_properties": {"data_size": 30433856, "index_size": 1770048, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26048148, "raw_average_key_size": 28, "raw_value_size": 108169678, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930291, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336016, "oldest_key_time": 0, "file_creation_time": 1732336016, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259434, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:26:59.797320 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259435.sst) /dev/shm/ToplingTemp/Topling-48109-Y7NRNY 2024/11/23-12:27:00.685631 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259435.sst): first pass time = 0.89's, 151.099'MB/sec 2024/11/23-12:27:02.205028 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259435.sst): old prealloc_size = 147639500, real_size = 36147552 2024/11/23-12:27:02.205577 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259435.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930581 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0341 GB avg-val = 116.23 avg-zval = 36.68 2024/11/23-12:27:02.206142 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259435.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:27:02.985387 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259435.sst): second pass time = 1.52's, 71.227'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 50.160'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3364.166'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.357'MB/sec write SST data time = 0.13's, 304.829'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 37066.992'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.227'MB/sec zip pipeline throughput = 71.227'MB/sec entries = 930581 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.23 avg-zval = 36.68 usrkeys = 930581 avg-key = 20.00 avg-zkey = 1.91 avg-val = 120.23 avg-zval = 36.68 TagRS{ kinds = RS_Key0_TagN bytes = 145520 } TagArray{ size = 531379 bytes = 1992688 } seq expand size = 3719653 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0341 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.4471 value = 3.1688 dict = 1.00 all = 3.4291 } Zip/UnZip{ index = 0.0957 value = 0.3156 dict = 1.00 all = 0.2916 } ---------------------------- total value len = 9.533759 GB avg = 0.165 KB (by entry num) total key len = 1.597319 GB avg = 0.028 KB total ukey len = 1.109662 GB avg = 0.016 KB total ukey num = 0.071139637 Billion total entry num = 0.057947366 Billion write speed all = 0.875720121 MB/sec (with seq num) write speed all = 0.839248774 MB/sec (without seq num) 2024/11/23-12:27:03.034067 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259435: 930581 keys, 39141126 bytes, temperature: kWarm 2024/11/23-12:27:03.034137 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336023034102, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259435, "file_size": 39141126, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3973691221, "table_properties": {"data_size": 34133200, "index_size": 1781504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26056268, "raw_average_key_size": 28, "raw_value_size": 108161578, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930581, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336019, "oldest_key_time": 0, "file_creation_time": 1732336019, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259435, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:27:03.074147 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259436.sst) /dev/shm/ToplingTemp/Topling-48109-ntzQUY 2024/11/23-12:27:03.988706 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259436.sst): first pass time = 0.91's, 146.764'MB/sec 2024/11/23-12:27:05.555260 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259436.sst): old prealloc_size = 147639500, real_size = 33976656 2024/11/23-12:27:05.555787 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259436.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928497 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0320 GB avg-val = 116.55 avg-zval = 34.44 2024/11/23-12:27:05.556410 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259436.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:27:06.374725 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259436.sst): second pass time = 1.57's, 69.134'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 45.478'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3498.574'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.337'MB/sec write SST data time = 0.13's, 286.460'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30020.690'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 69.134'MB/sec zip pipeline throughput = 103.619'MB/sec entries = 928497 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.55 avg-zval = 34.44 usrkeys = 928497 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.15 avg-zval = 34.44 TagRS{ kinds = RS_Key0_TagN bytes = 145160 } TagArray{ size = 610219 bytes = 2288336 } seq expand size = 4271533 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5121 value = 3.3842 dict = 1.00 all = 3.6015 } Zip/UnZip{ index = 0.0951 value = 0.2955 dict = 1.00 all = 0.2777 } ---------------------------- total value len = 9.646250 GB avg = 0.164 KB (by entry num) total key len = 1.623317 GB avg = 0.028 KB total ukey len = 1.128232 GB avg = 0.016 KB total ukey num = 0.072068134 Billion total entry num = 0.058875863 Billion write speed all = 0.886379200 MB/sec (with seq num) write speed all = 0.849333348 MB/sec (without seq num) 2024/11/23-12:27:06.420866 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259436: 928497 keys, 37267144 bytes, temperature: kWarm 2024/11/23-12:27:06.420951 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336026420901, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259436, "file_size": 37267144, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3973691444, "table_properties": {"data_size": 31977792, "index_size": 1766528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25997916, "raw_average_key_size": 28, "raw_value_size": 108219918, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928497, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336023, "oldest_key_time": 0, "file_creation_time": 1732336023, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259436, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:27:06.457417 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259437.sst) /dev/shm/ToplingTemp/Topling-48109-fIF6M0 2024/11/23-12:27:07.440944 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259437.sst): first pass time = 0.98's, 136.471'MB/sec 2024/11/23-12:27:09.000877 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259437.sst): old prealloc_size = 147639500, real_size = 32298496 2024/11/23-12:27:09.001446 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259437.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933620 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0303 GB avg-val = 115.76 avg-zval = 32.43 2024/11/23-12:27:09.002258 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259437.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:27:09.864446 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259437.sst): second pass time = 1.56's, 69.314'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 42.306'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3256.765'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 281.092'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 39076.076'MB/sec zip my value time = 1.56's, unzip length = 0.108'GB zip my value throughput = 69.314'MB/sec zip pipeline throughput = 123.484'MB/sec entries = 933620 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.76 avg-zval = 32.43 usrkeys = 933620 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.76 avg-zval = 32.43 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933620 bytes = 3384384 } seq expand size = 6535340 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.4464 value = 3.5695 dict = 1.00 all = 3.6738 } Zip/UnZip{ index = 0.0957 value = 0.2801 dict = 1.00 all = 0.2722 } ---------------------------- total value len = 9.760862 GB avg = 0.163 KB (by entry num) total key len = 1.649458 GB avg = 0.028 KB total ukey len = 1.146904 GB avg = 0.016 KB total ukey num = 0.073001754 Billion total entry num = 0.059809483 Billion write speed all = 0.897203523 MB/sec (with seq num) write speed all = 0.859580545 MB/sec (without seq num) 2024/11/23-12:27:09.908353 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259437: 933620 keys, 36533460 bytes, temperature: kWarm 2024/11/23-12:27:09.908420 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336029908386, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259437, "file_size": 36533460, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3509327862, "largest_seqno": 3973691592, "table_properties": {"data_size": 30277424, "index_size": 1787456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26141360, "raw_average_key_size": 28, "raw_value_size": 108076464, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933620, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336026, "oldest_key_time": 0, "file_creation_time": 1732336026, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259437, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:27:09.946561 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259438.sst) /dev/shm/ToplingTemp/Topling-48109-kO63e0 2024/11/23-12:27:10.784387 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259438.sst): first pass time = 0.84's, 160.204'MB/sec 2024/11/23-12:27:12.369363 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259438.sst): old prealloc_size = 147639500, real_size = 32784128 2024/11/23-12:27:12.369910 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259438.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926759 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0308 GB avg-val = 116.82 avg-zval = 33.22 2024/11/23-12:27:12.370703 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259438.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:27:13.231479 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259438.sst): second pass time = 1.58's, 68.341'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 42.916'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3428.543'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 282.048'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30429.175'MB/sec zip my value time = 1.58's, unzip length = 0.108'GB zip my value throughput = 68.341'MB/sec zip pipeline throughput = 132.131'MB/sec entries = 926759 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.82 avg-zval = 33.22 usrkeys = 926759 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.82 avg-zval = 33.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926759 bytes = 3359520 } seq expand size = 6487313 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5134 value = 3.5164 dict = 1.00 all = 3.6277 } Zip/UnZip{ index = 0.0951 value = 0.2844 dict = 1.00 all = 0.2757 } ---------------------------- total value len = 9.875618 GB avg = 0.163 KB (by entry num) total key len = 1.675407 GB avg = 0.028 KB total ukey len = 1.165440 GB avg = 0.016 KB total ukey num = 0.073928513 Billion total entry num = 0.060736242 Billion write speed all = 0.908026884 MB/sec (with seq num) write speed all = 0.869831043 MB/sec (without seq num) 2024/11/23-12:27:13.295563 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259438: 926759 keys, 36997890 bytes, temperature: kWarm 2024/11/23-12:27:13.295608 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336033295581, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259438, "file_size": 36997890, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3650538033, "largest_seqno": 3973691487, "table_properties": {"data_size": 30789232, "index_size": 1763008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25949252, "raw_average_key_size": 28, "raw_value_size": 108268502, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926759, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336029, "oldest_key_time": 0, "file_creation_time": 1732336029, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259438, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:27:13.339145 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259439.sst) /dev/shm/ToplingTemp/Topling-48109-7lS8mW 2024/11/23-12:27:14.437263 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259439.sst): first pass time = 1.10's, 122.229'MB/sec 2024/11/23-12:27:15.761442 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259439.sst): old prealloc_size = 147639500, real_size = 32885616 2024/11/23-12:27:15.762032 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259439.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930711 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.21 avg-zval = 33.16 2024/11/23-12:27:15.764752 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259439.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:27:16.624951 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259439.sst): second pass time = 1.32's, 81.727'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 42.978'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3182.599'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 287.543'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 35598.891'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.727'MB/sec zip pipeline throughput = 81.727'MB/sec entries = 930711 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.21 avg-zval = 33.16 usrkeys = 930711 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.21 avg-zval = 33.16 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930711 bytes = 3373840 } seq expand size = 6514977 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.3767 value = 3.5049 dict = 1.00 all = 3.6167 } Zip/UnZip{ index = 0.0964 value = 0.2853 dict = 1.00 all = 0.2765 } ---------------------------- total value len = 9.990291 GB avg = 0.162 KB (by entry num) total key len = 1.701467 GB avg = 0.028 KB total ukey len = 1.184054 GB avg = 0.016 KB total ukey num = 0.074859224 Billion total entry num = 0.061666953 Billion write speed all = 0.918844783 MB/sec (with seq num) write speed all = 0.880073979 MB/sec (without seq num) 2024/11/23-12:27:16.670086 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259439: 930711 keys, 37110740 bytes, temperature: kWarm 2024/11/23-12:27:16.670155 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336036670119, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259439, "file_size": 37110740, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3583362754, "largest_seqno": 3973691655, "table_properties": {"data_size": 30858912, "index_size": 1793856, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26059908, "raw_average_key_size": 28, "raw_value_size": 108157858, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930711, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336033, "oldest_key_time": 0, "file_creation_time": 1732336033, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259439, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:27:16.713690 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259440.sst) /dev/shm/ToplingTemp/Topling-48109-9D8BFY 2024/11/23-12:27:17.743584 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259440.sst): first pass time = 1.03's, 130.327'MB/sec 2024/11/23-12:27:18.994246 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259440.sst): old prealloc_size = 147639500, real_size = 31729936 2024/11/23-12:27:18.994831 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259440.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 937431 prefix = 4 raw-key = 0.0150 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1080 GB zip-val = 0.0297 GB avg-val = 115.18 avg-zval = 31.68 2024/11/23-12:27:18.996828 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259440.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-12:27:19.856219 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259440.sst): second pass time = 1.25's, 86.388'MB/sec, value only(80.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 41.738'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3190.126'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.321'MB/sec write SST data time = 0.13's, 275.403'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.077'MB, throughput = 35413.092'MB/sec zip my value time = 1.25's, unzip length = 0.108'GB zip my value throughput = 86.388'MB/sec zip pipeline throughput = 86.388'MB/sec entries = 937431 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.18 avg-zval = 31.68 usrkeys = 937431 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.18 avg-zval = 31.68 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 937431 bytes = 3398208 } seq expand size = 6562017 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0187 GB value = 0.1080 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0297 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.4460 value = 3.6353 dict = 1.00 all = 3.7307 } Zip/UnZip{ index = 0.0957 value = 0.2751 dict = 1.00 all = 0.2680 } ---------------------------- total value len = 10.104823 GB avg = 0.161 KB (by entry num) total key len = 1.727715 GB avg = 0.028 KB total ukey len = 1.202802 GB avg = 0.016 KB total ukey num = 0.075796655 Billion total entry num = 0.062604384 Billion write speed all = 0.929672458 MB/sec (with seq num) write speed all = 0.890322271 MB/sec (without seq num) 2024/11/23-12:27:19.900455 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259440: 937431 keys, 35976350 bytes, temperature: kWarm 2024/11/23-12:27:19.900523 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336039900489, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259440, "file_size": 35976350, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3621075956, "largest_seqno": 3973243363, "table_properties": {"data_size": 29700608, "index_size": 1794816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26248068, "raw_average_key_size": 28, "raw_value_size": 107969710, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 937431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336036, "oldest_key_time": 0, "file_creation_time": 1732336036, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259440, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:27:19.937582 139957240342080 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259441.sst) /dev/shm/ToplingTemp/Topling-48109-eu5DXZ 2024/11/23-12:27:20.051874 139957240342080 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259441.sst): first pass time = 0.11's, 239.906'MB/sec 2024/11/23-12:27:20.315318 139957240342080 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259441.sst): old prealloc_size = 147639500, real_size = 7982256 2024/11/23-12:27:20.315487 139957240342080 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259441.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 189864 prefix = 4 raw-key = 0.0030 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0221 GB zip-val = 0.0076 GB avg-val = 116.37 avg-zval = 39.87 2024/11/23-12:27:20.315688 139957240342080 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259441.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.027 GB 2024/11/23-12:27:20.479798 139957240342080 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00/259441.sst): second pass time = 0.26's, 84.085'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.16's, 54.000'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2478.208'MB/sec (index lex order gen) rebuild zvType time = 0.14's, 0.344'MB/sec write SST data time = 0.03's, 337.723'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.232'MB, throughput = 7913.311'MB/sec zip my value time = 0.26's, unzip length = 0.022'GB zip my value throughput = 84.085'MB/sec zip pipeline throughput = 84.085'MB/sec entries = 189864 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.37 avg-zval = 39.87 usrkeys = 189864 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.37 avg-zval = 39.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 189864 bytes = 712000 } seq expand size = 1329048 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0038 GB value = 0.0221 GB dict = 0.23 MB all = 0.0274 GB } __ZipSize{ index = 0.0004 GB value = 0.0076 GB dict = 0.23 MB all = 0.0089 GB } UnZip/Zip{ index = 10.4183 value = 2.9187 dict = 1.00 all = 3.0865 } Zip/UnZip{ index = 0.0960 value = 0.3426 dict = 1.00 all = 0.3240 } ---------------------------- total value len = 10.128246 GB avg = 0.161 KB (by entry num) total key len = 1.733031 GB avg = 0.028 KB total ukey len = 1.206600 GB avg = 0.016 KB total ukey num = 0.075986519 Billion total entry num = 0.062794248 Billion write speed all = 0.931884872 MB/sec (with seq num) write speed all = 0.892417279 MB/sec (without seq num) 2024/11/23-12:27:20.496513 139957240342080 [compaction_job.cc:1948] [default] [JOB 25975] Generated table #259441: 189864 keys, 8880771 bytes, temperature: kWarm 2024/11/23-12:27:20.496572 139957240342080 EVENT_LOG_v1 {"time_micros": 1732336040496543, "cf_name": "default", "job": 25975, "event": "table_file_creation", "file_number": 259441, "file_size": 8880771, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3972196173, "table_properties": {"data_size": 7570144, "index_size": 364480, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5316192, "raw_average_key_size": 28, "raw_value_size": 22094676, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 189864, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732336039, "oldest_key_time": 0, "file_creation_time": 1732336039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 259441, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-12:27:20.510714 139957240342080 [compaction_job.cc:703] job-25975: subcompact[0], size: 0.413814 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-25975/att-00 [259430,259431,259432,259433,259434,259435,259436,259437,259438,259439,259440,259441] 2024/11/23-12:27:20.584395 139957240342080 (Original Log Time 2024/11/23-12:26:36.305402) EVENT_LOG_v1 {"time_micros": 1732335996305354, "job": 25975, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [259413], "files_L3": [259356, 259357, 259358, 259359, 259360, 259361, 259362, 259363, 259364, 259365, 259367], "score": 0.999621, "input_data_size": 432383547, "oldest_snapshot_seqno": 3973743975} 2024/11/23-12:27:20.584398 139957240342080 (Original Log Time 2024/11/23-12:27:20.583384) [compaction_job.cc:2022] [default] [JOB 25975] Compacted 1@2 + 11@3 files to L3 => 413813713 bytes 2024/11/23-12:27:20.584399 139957240342080 (Original Log Time 2024/11/23-12:27:20.584325) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 9.8 rd, 9.4 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.1, 385.2 +0.0 blob) out(394.6 +0.0 blob), read-write-amplify(29.8) write-amplify(14.6) OK, records in: 10479287, records dropped: 43203 output_compression: Snappy 2024/11/23-12:27:20.584404 139957240342080 (Original Log Time 2024/11/23-12:27:20.584367) EVENT_LOG_v1 {"time_micros": 1732336040584347, "cf": "default", "job": 25975, "event": "compaction_finished", "compaction_time_micros": 44205319, "compaction_time_cpu_micros": 17611311, "output_level": 3, "num_output_files": 12, "total_output_size": 413813713, "num_input_records": 10479287, "num_output_records": 10436084, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 300710733, "file_range_sync_nanos": 0, "file_fsync_nanos": 8804, "file_prepare_write_nanos": 31738, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/23-12:27:20.709397 139957240342080 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732335996,"executesMs":44407,"compactionJobId":25975,"attempt":0,"compactionInputRawBytes":1509941458,"compactionInputZipBytes":432383547,"compactionOutputRawBytes":0,"compactionOutputZipBytes":413813713,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-fd8b37890aa49035ae7943e4e3d5c8ff-791f39aabf8e7bc4-00"}}, response = 2024/11/23-12:27:20.709485 139957240342080 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-25975/att-00: olev 3, work 44.407 s, result 7.613 ms, install 1.042 ms, input{raw 1.406 GiB zip 412.353 MiB}