2024/11/24-02:38:57.906689 139957643191872 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00: fork to child time = 0.021721 sec 2024/11/24-02:38:57.907706 139957643191872 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00 2024/11/24-02:38:57.914782 139957643191872 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00 2024/11/24-02:38:57.916018 139957643191872 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-02:38:57.916345 139957643191872 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-02:38:57.970468 139957643191872 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40235/att-00/MANIFEST-206347 2024/11/24-02:38:58.002443 139957643191872 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40235/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 289078, last_sequence is 4399487159, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 288909 2024/11/24-02:38:58.002464 139957643191872 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-02:38:58.002674 139957643191872 [version_set.cc:5721] Creating manifest 289078 2024/11/24-02:38:58.038758 139957643191872 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-02:38:58.060311 139957643191872 [compaction_job.cc:2437] [default] [JOB 40235] Compacting 1@2 + 18@3 files to L3, score 0.61, subcompactions 1 : 1 2024/11/24-02:38:58.060335 139957643191872 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 30138 Base level 2, inputs: [289034(27MB)], [289069(2054KB) 288993(33MB) 288994(34MB) 288995(35MB) 288996(35MB) 288997(35MB) 288998(30MB) 288999(35MB) 289000(31MB) 289001(35MB) 289002(32MB) 289003(33MB) 289004(34MB) 289005(11MB) 288962(32MB) 288963(33MB) 288964(35MB) 289070(33MB)] 2024/11/24-02:38:58.068893 139957643191872 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-02:38:58.068923 139957643191872 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-02:38:58.068950 139957643191872 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-02:38:58.068955 139957643191872 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-02:38:58.068967 139957643191872 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-02:38:58.403015 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289079.sst) /dev/shm/ToplingTemp/Topling-250155-ngNt9L 2024/11/24-02:38:59.400342 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289079.sst): first pass time = 1.00's, 134.584'MB/sec 2024/11/24-02:39:00.780772 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289079.sst): old prealloc_size = 147639500, real_size = 31367408 2024/11/24-02:39:00.781398 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289079.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921877 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0294 GB avg-val = 117.59 avg-zval = 31.87 2024/11/24-02:39:00.782199 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289079.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:01.706983 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289079.sst): second pass time = 1.38's, 78.617'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 38.279'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2986.697'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.288'MB/sec write SST data time = 0.13's, 280.074'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 36032.057'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.617'MB/sec zip pipeline throughput = 78.617'MB/sec entries = 921877 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.59 avg-zval = 31.87 usrkeys = 921877 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.59 avg-zval = 31.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921877 bytes = 3226592 } seq expand size = 6453139 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.08 MB all = 0.0355 GB } UnZip/Zip{ index = 10.5149 value = 3.6894 dict = 1.00 all = 3.7858 } Zip/UnZip{ index = 0.0951 value = 0.2711 dict = 1.00 all = 0.2641 } ---------------------------- total value len = 2251.440432 GB avg = 0.200 KB (by entry num) total key len = 306.908775 GB avg = 0.027 KB total ukey len = 237.373165 GB avg = 0.015 KB total ukey num = 15.739583040 Billion total entry num = 11.280896988 Billion write speed all = 40.080989429 MB/sec (with seq num) write speed all = 38.667110474 MB/sec (without seq num) 2024/11/24-02:39:01.751261 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289079: 921877 keys, 35452880 bytes, temperature: kWarm 2024/11/24-02:39:01.751362 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387141751299, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289079, "file_size": 35452880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178730, "largest_seqno": 4399306524, "table_properties": {"data_size": 29383264, "index_size": 1753472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25812556, "raw_average_key_size": 28, "raw_value_size": 108405294, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921877, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387138, "oldest_key_time": 0, "file_creation_time": 1732387138, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289079, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:01.789565 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289080.sst) /dev/shm/ToplingTemp/Topling-250155-aB3aFJ 2024/11/24-02:39:02.800472 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289080.sst): first pass time = 1.01's, 132.776'MB/sec 2024/11/24-02:39:04.159154 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289080.sst): old prealloc_size = 147639500, real_size = 32746624 2024/11/24-02:39:04.159711 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289080.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923931 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0308 GB avg-val = 117.27 avg-zval = 33.29 2024/11/24-02:39:04.160498 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289080.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:05.104929 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289080.sst): second pass time = 1.36's, 79.793'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 38.828'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3283.949'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.282'MB/sec write SST data time = 0.13's, 293.173'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 38107.922'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.793'MB/sec zip pipeline throughput = 79.793'MB/sec entries = 923931 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.27 avg-zval = 33.29 usrkeys = 923931 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.27 avg-zval = 33.29 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923931 bytes = 3118288 } seq expand size = 6467517 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5126 value = 3.5226 dict = 1.00 all = 3.6550 } Zip/UnZip{ index = 0.0951 value = 0.2839 dict = 1.00 all = 0.2736 } ---------------------------- total value len = 2251.555247 GB avg = 0.200 KB (by entry num) total key len = 306.934645 GB avg = 0.027 KB total ukey len = 237.391643 GB avg = 0.015 KB total ukey num = 15.740506971 Billion total entry num = 11.281820919 Billion write speed all = 40.081059795 MB/sec (with seq num) write speed all = 38.667140310 MB/sec (without seq num) 2024/11/24-02:39:05.150011 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289080: 923931 keys, 36721682 bytes, temperature: kWarm 2024/11/24-02:39:05.150160 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387145150124, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289080, "file_size": 36721682, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265680186, "largest_seqno": 4399305279, "table_properties": {"data_size": 30757680, "index_size": 1757760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25870068, "raw_average_key_size": 28, "raw_value_size": 108347722, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923931, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387141, "oldest_key_time": 0, "file_creation_time": 1732387141, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289080, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:05.188091 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289081.sst) /dev/shm/ToplingTemp/Topling-250155-L6v0XK 2024/11/24-02:39:06.220513 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289081.sst): first pass time = 1.03's, 130.007'MB/sec 2024/11/24-02:39:07.583130 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289081.sst): old prealloc_size = 147639500, real_size = 33409104 2024/11/24-02:39:07.583645 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289081.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926388 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.88 avg-zval = 33.91 2024/11/24-02:39:07.584386 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289081.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:08.538455 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289081.sst): second pass time = 1.36's, 79.530'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 39.139'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3598.461'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.279'MB/sec write SST data time = 0.13's, 297.980'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34004.219'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.530'MB/sec zip pipeline throughput = 79.530'MB/sec entries = 926388 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.88 avg-zval = 33.91 usrkeys = 926388 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.88 avg-zval = 33.91 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926388 bytes = 3126576 } seq expand size = 6484716 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5111 value = 3.4468 dict = 1.00 all = 3.5897 } Zip/UnZip{ index = 0.0951 value = 0.2901 dict = 1.00 all = 0.2786 } ---------------------------- total value len = 2251.670010 GB avg = 0.200 KB (by entry num) total key len = 306.960584 GB avg = 0.027 KB total ukey len = 237.410171 GB avg = 0.015 KB total ukey num = 15.741433359 Billion total entry num = 11.282747307 Billion write speed all = 40.081108090 MB/sec (with seq num) write speed all = 38.667148559 MB/sec (without seq num) 2024/11/24-02:39:08.584872 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289081: 926388 keys, 37390014 bytes, temperature: kWarm 2024/11/24-02:39:08.584948 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387148584906, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289081, "file_size": 37390014, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265680199, "largest_seqno": 4399306535, "table_properties": {"data_size": 31414656, "index_size": 1762688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25938864, "raw_average_key_size": 28, "raw_value_size": 108278864, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926388, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387145, "oldest_key_time": 0, "file_creation_time": 1732387145, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289081, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:08.622999 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289082.sst) /dev/shm/ToplingTemp/Topling-250155-l46BPJ 2024/11/24-02:39:09.645440 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289082.sst): first pass time = 1.02's, 131.278'MB/sec 2024/11/24-02:39:11.043679 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289082.sst): old prealloc_size = 147639500, real_size = 33553584 2024/11/24-02:39:11.044261 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289082.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926164 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0316 GB avg-val = 116.92 avg-zval = 34.08 2024/11/24-02:39:11.045019 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289082.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:12.011714 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289082.sst): second pass time = 1.40's, 77.484'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 38.775'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3160.044'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.275'MB/sec write SST data time = 0.13's, 295.715'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 27668.195'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.484'MB/sec zip pipeline throughput = 77.484'MB/sec entries = 926164 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.92 avg-zval = 34.08 usrkeys = 926164 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.92 avg-zval = 34.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926164 bytes = 3125824 } seq expand size = 6483148 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5158 value = 3.4310 dict = 1.00 all = 3.5759 } Zip/UnZip{ index = 0.0951 value = 0.2915 dict = 1.00 all = 0.2797 } ---------------------------- total value len = 2251.784779 GB avg = 0.200 KB (by entry num) total key len = 306.986517 GB avg = 0.027 KB total ukey len = 237.428695 GB avg = 0.015 KB total ukey num = 15.742359523 Billion total entry num = 11.283673471 Billion write speed all = 40.081131414 MB/sec (with seq num) write speed all = 38.667132749 MB/sec (without seq num) 2024/11/24-02:39:12.057328 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289082: 926164 keys, 37534302 bytes, temperature: kWarm 2024/11/24-02:39:12.057384 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387152057353, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289082, "file_size": 37534302, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265680212, "largest_seqno": 4399306548, "table_properties": {"data_size": 31560416, "index_size": 1761472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25932592, "raw_average_key_size": 28, "raw_value_size": 108285180, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926164, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387148, "oldest_key_time": 0, "file_creation_time": 1732387148, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289082, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:12.095041 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289083.sst) /dev/shm/ToplingTemp/Topling-250155-LqQO0I 2024/11/24-02:39:13.132282 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289083.sst): first pass time = 1.04's, 129.403'MB/sec 2024/11/24-02:39:14.546278 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289083.sst): old prealloc_size = 147639500, real_size = 33526688 2024/11/24-02:39:14.546853 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289083.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926442 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.87 avg-zval = 34.02 2024/11/24-02:39:14.547637 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289083.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:15.505323 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289083.sst): second pass time = 1.41's, 76.618'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 39.114'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3217.708'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.279'MB/sec write SST data time = 0.13's, 294.328'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27478.433'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.618'MB/sec zip pipeline throughput = 76.618'MB/sec entries = 926442 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.87 avg-zval = 34.02 usrkeys = 926442 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.87 avg-zval = 34.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926442 bytes = 3126752 } seq expand size = 6485094 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4442 value = 3.4351 dict = 1.00 all = 3.5780 } Zip/UnZip{ index = 0.0957 value = 0.2911 dict = 1.00 all = 0.2795 } ---------------------------- total value len = 2251.899541 GB avg = 0.200 KB (by entry num) total key len = 307.012457 GB avg = 0.027 KB total ukey len = 237.447223 GB avg = 0.015 KB total ukey num = 15.743285965 Billion total entry num = 11.284599913 Billion write speed all = 40.081141988 MB/sec (with seq num) write speed all = 38.667104610 MB/sec (without seq num) 2024/11/24-02:39:15.550931 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289083: 926442 keys, 37511636 bytes, temperature: kWarm 2024/11/24-02:39:15.551034 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387155550988, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289083, "file_size": 37511636, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679834, "largest_seqno": 4399306568, "table_properties": {"data_size": 31520784, "index_size": 1774080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25940376, "raw_average_key_size": 28, "raw_value_size": 108277452, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926442, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387152, "oldest_key_time": 0, "file_creation_time": 1732387152, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289083, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:15.586792 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289084.sst) /dev/shm/ToplingTemp/Topling-250155-Vw7KrM 2024/11/24-02:39:16.454686 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289084.sst): first pass time = 0.87's, 154.654'MB/sec 2024/11/24-02:39:17.754101 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289084.sst): old prealloc_size = 147639500, real_size = 28567632 2024/11/24-02:39:17.754615 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289084.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 941158 prefix = 4 raw-key = 0.0151 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1079 GB zip-val = 0.0265 GB avg-val = 114.61 avg-zval = 28.19 2024/11/24-02:39:17.755404 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289084.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0024 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:18.691195 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289084.sst): second pass time = 1.30's, 83.061'MB/sec, value only(80.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 34.778'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3694.210'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.292'MB/sec write SST data time = 0.13's, 250.193'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.076'MB, throughput = 36160.954'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.061'MB/sec zip pipeline throughput = 83.061'MB/sec entries = 941158 avg-key = 28.00 avg-zkey = 1.91 avg-val = 114.61 avg-zval = 28.19 usrkeys = 941158 avg-key = 20.00 avg-zkey = 1.91 avg-val = 121.61 avg-zval = 28.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 941158 bytes = 3176416 } seq expand size = 6588106 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0188 GB value = 0.1079 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0265 GB dict = 1.08 MB all = 0.0326 GB } UnZip/Zip{ index = 10.4447 value = 4.0658 dict = 1.00 all = 4.1185 } Zip/UnZip{ index = 0.0957 value = 0.2460 dict = 1.00 all = 0.2428 } ---------------------------- total value len = 2252.013995 GB avg = 0.200 KB (by entry num) total key len = 307.038809 GB avg = 0.027 KB total ukey len = 237.466047 GB avg = 0.015 KB total ukey num = 15.744227123 Billion total entry num = 11.285541071 Billion write speed all = 40.081347365 MB/sec (with seq num) write speed all = 38.667262619 MB/sec (without seq num) 2024/11/24-02:39:18.733240 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289084: 941158 keys, 32588934 bytes, temperature: kWarm 2024/11/24-02:39:18.733312 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387158733276, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289084, "file_size": 32588934, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265680463, "largest_seqno": 4399305667, "table_properties": {"data_size": 26529984, "index_size": 1802176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26352424, "raw_average_key_size": 28, "raw_value_size": 107865304, "raw_average_value_size": 114, "num_data_blocks": 1, "num_entries": 941158, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387155, "oldest_key_time": 0, "file_creation_time": 1732387155, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289084, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:18.778681 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289085.sst) /dev/shm/ToplingTemp/Topling-250155-0UKGuJ 2024/11/24-02:39:19.724180 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289085.sst): first pass time = 0.95's, 141.963'MB/sec 2024/11/24-02:39:21.094192 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289085.sst): old prealloc_size = 147639500, real_size = 32708640 2024/11/24-02:39:21.094764 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289085.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931843 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0307 GB avg-val = 116.03 avg-zval = 32.95 2024/11/24-02:39:21.095550 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289085.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:22.067398 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289085.sst): second pass time = 1.37's, 79.002'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 37.716'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3220.970'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.276'MB/sec write SST data time = 0.13's, 282.578'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27297.077'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.002'MB/sec zip pipeline throughput = 79.002'MB/sec entries = 931843 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.03 avg-zval = 32.95 usrkeys = 931843 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.03 avg-zval = 32.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931843 bytes = 3144992 } seq expand size = 6522901 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5119 value = 3.5217 dict = 1.00 all = 3.6568 } Zip/UnZip{ index = 0.0951 value = 0.2840 dict = 1.00 all = 0.2735 } ---------------------------- total value len = 2252.128644 GB avg = 0.200 KB (by entry num) total key len = 307.064901 GB avg = 0.027 KB total ukey len = 237.484683 GB avg = 0.015 KB total ukey num = 15.745158966 Billion total entry num = 11.286472914 Billion write speed all = 40.081432219 MB/sec (with seq num) write speed all = 38.667305492 MB/sec (without seq num) 2024/11/24-02:39:22.112193 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289085: 931843 keys, 36704082 bytes, temperature: kWarm 2024/11/24-02:39:22.112260 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387162112225, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289085, "file_size": 36704082, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679848, "largest_seqno": 4399304025, "table_properties": {"data_size": 30702544, "index_size": 1772928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26091604, "raw_average_key_size": 28, "raw_value_size": 108126230, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931843, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387158, "oldest_key_time": 0, "file_creation_time": 1732387158, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289085, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:22.155530 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289086.sst) /dev/shm/ToplingTemp/Topling-250155-0ofuDI 2024/11/24-02:39:23.152968 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289086.sst): first pass time = 1.00's, 134.567'MB/sec 2024/11/24-02:39:24.480565 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289086.sst): old prealloc_size = 147639500, real_size = 29703488 2024/11/24-02:39:24.481181 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289086.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 934281 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0277 GB avg-val = 115.66 avg-zval = 29.63 2024/11/24-02:39:24.481969 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289086.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:25.426863 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289086.sst): second pass time = 1.33's, 81.442'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 35.620'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3030.979'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.287'MB/sec write SST data time = 0.13's, 256.964'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 38570.376'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.442'MB/sec zip pipeline throughput = 81.442'MB/sec entries = 934281 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.66 avg-zval = 29.63 usrkeys = 934281 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.66 avg-zval = 29.63 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 934281 bytes = 3153216 } seq expand size = 6539967 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0277 GB dict = 1.08 MB all = 0.0337 GB } UnZip/Zip{ index = 10.4440 value = 3.9037 dict = 1.00 all = 3.9820 } Zip/UnZip{ index = 0.0957 value = 0.2562 dict = 1.00 all = 0.2511 } ---------------------------- total value len = 2252.243242 GB avg = 0.200 KB (by entry num) total key len = 307.091061 GB avg = 0.027 KB total ukey len = 237.503369 GB avg = 0.015 KB total ukey num = 15.746093247 Billion total entry num = 11.287407195 Billion write speed all = 40.081527839 MB/sec (with seq num) write speed all = 38.667358458 MB/sec (without seq num) 2024/11/24-02:39:25.469812 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289086: 934281 keys, 33705734 bytes, temperature: kWarm 2024/11/24-02:39:25.469883 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387165469848, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289086, "file_size": 33705734, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4298050943, "largest_seqno": 4399306299, "table_properties": {"data_size": 27680624, "index_size": 1789120, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26159868, "raw_average_key_size": 28, "raw_value_size": 108057974, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 934281, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387162, "oldest_key_time": 0, "file_creation_time": 1732387162, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289086, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:25.507827 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289087.sst) /dev/shm/ToplingTemp/Topling-250155-1NpaJM 2024/11/24-02:39:26.370160 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289087.sst): first pass time = 0.86's, 155.652'MB/sec 2024/11/24-02:39:27.753776 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289087.sst): old prealloc_size = 147639500, real_size = 32945648 2024/11/24-02:39:27.754344 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289087.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930127 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.30 avg-zval = 33.27 2024/11/24-02:39:27.755134 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289087.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:28.734708 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289087.sst): second pass time = 1.38's, 78.254'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 37.656'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3257.072'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.274'MB/sec write SST data time = 0.13's, 283.570'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 36694.634'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.254'MB/sec zip pipeline throughput = 78.254'MB/sec entries = 930127 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.30 avg-zval = 33.27 usrkeys = 930127 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.30 avg-zval = 33.27 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930127 bytes = 3139200 } seq expand size = 6510889 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5123 value = 3.4959 dict = 1.00 all = 3.6337 } Zip/UnZip{ index = 0.0951 value = 0.2861 dict = 1.00 all = 0.2752 } ---------------------------- total value len = 2252.357927 GB avg = 0.200 KB (by entry num) total key len = 307.117104 GB avg = 0.027 KB total ukey len = 237.521972 GB avg = 0.015 KB total ukey num = 15.747023374 Billion total entry num = 11.288337322 Billion write speed all = 40.081655391 MB/sec (with seq num) write speed all = 38.667442739 MB/sec (without seq num) 2024/11/24-02:39:28.780819 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289087: 930127 keys, 36936684 bytes, temperature: kWarm 2024/11/24-02:39:28.780890 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387168780853, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289087, "file_size": 36936684, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679865, "largest_seqno": 4399304884, "table_properties": {"data_size": 30943328, "index_size": 1769600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26043556, "raw_average_key_size": 28, "raw_value_size": 108174222, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930127, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387165, "oldest_key_time": 0, "file_creation_time": 1732387165, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289087, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:28.817877 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289088.sst) /dev/shm/ToplingTemp/Topling-250155-4CWe3I 2024/11/24-02:39:29.829883 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289088.sst): first pass time = 1.01's, 132.630'MB/sec 2024/11/24-02:39:31.170972 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289088.sst): old prealloc_size = 147639500, real_size = 30153344 2024/11/24-02:39:31.171562 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289088.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930488 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0281 GB avg-val = 116.24 avg-zval = 30.24 2024/11/24-02:39:31.172374 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289088.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:32.110222 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289088.sst): second pass time = 1.34's, 80.702'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 36.353'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3223.251'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.288'MB/sec write SST data time = 0.13's, 261.236'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33068.682'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.702'MB/sec zip pipeline throughput = 80.702'MB/sec entries = 930488 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.24 avg-zval = 30.24 usrkeys = 930488 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.24 avg-zval = 30.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930488 bytes = 3140416 } seq expand size = 6513416 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0281 GB dict = 1.08 MB all = 0.0341 GB } UnZip/Zip{ index = 10.4457 value = 3.8439 dict = 1.00 all = 3.9310 } Zip/UnZip{ index = 0.0957 value = 0.2602 dict = 1.00 all = 0.2544 } ---------------------------- total value len = 2252.472605 GB avg = 0.200 KB (by entry num) total key len = 307.143158 GB avg = 0.027 KB total ukey len = 237.540581 GB avg = 0.015 KB total ukey num = 15.747953862 Billion total entry num = 11.289267810 Billion write speed all = 40.081740503 MB/sec (with seq num) write speed all = 38.667486037 MB/sec (without seq num) 2024/11/24-02:39:32.152660 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289088: 930488 keys, 34143670 bytes, temperature: kWarm 2024/11/24-02:39:32.152735 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387172152698, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289088, "file_size": 34143670, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4332050627, "largest_seqno": 4399306318, "table_properties": {"data_size": 28138992, "index_size": 1781568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26053664, "raw_average_key_size": 28, "raw_value_size": 108164184, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930488, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387168, "oldest_key_time": 0, "file_creation_time": 1732387168, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289088, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:32.189706 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289089.sst) /dev/shm/ToplingTemp/Topling-250155-bnEyrI 2024/11/24-02:39:33.208597 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289089.sst): first pass time = 1.02's, 131.734'MB/sec 2024/11/24-02:39:34.538891 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289089.sst): old prealloc_size = 147639500, real_size = 32577408 2024/11/24-02:39:34.539482 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289089.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922825 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0306 GB avg-val = 117.44 avg-zval = 33.15 2024/11/24-02:39:34.540220 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289089.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:35.462580 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289089.sst): second pass time = 1.33's, 81.550'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 39.571'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3085.129'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.291'MB/sec write SST data time = 0.13's, 278.153'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 35788.120'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.550'MB/sec zip pipeline throughput = 81.550'MB/sec entries = 922825 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.44 avg-zval = 33.15 usrkeys = 922825 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.44 avg-zval = 33.15 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922825 bytes = 3114544 } seq expand size = 6459775 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.5126 value = 3.5428 dict = 1.00 all = 3.6722 } Zip/UnZip{ index = 0.0951 value = 0.2823 dict = 1.00 all = 0.2723 } ---------------------------- total value len = 2252.587443 GB avg = 0.200 KB (by entry num) total key len = 307.168997 GB avg = 0.027 KB total ukey len = 237.559038 GB avg = 0.015 KB total ukey num = 15.748876687 Billion total entry num = 11.290190635 Billion write speed all = 40.081839296 MB/sec (with seq num) write speed all = 38.667543469 MB/sec (without seq num) 2024/11/24-02:39:35.507218 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289089: 922825 keys, 36549786 bytes, temperature: kWarm 2024/11/24-02:39:35.507287 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387175507252, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289089, "file_size": 36549786, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679880, "largest_seqno": 4399306724, "table_properties": {"data_size": 30590896, "index_size": 1755648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25839100, "raw_average_key_size": 28, "raw_value_size": 108378762, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922825, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387172, "oldest_key_time": 0, "file_creation_time": 1732387172, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289089, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:35.544865 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289090.sst) /dev/shm/ToplingTemp/Topling-250155-FD5HzL 2024/11/24-02:39:36.581465 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289090.sst): first pass time = 1.04's, 129.483'MB/sec 2024/11/24-02:39:38.098186 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289090.sst): old prealloc_size = 147639500, real_size = 31977936 2024/11/24-02:39:38.098714 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289090.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930976 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0300 GB avg-val = 116.17 avg-zval = 32.20 2024/11/24-02:39:38.099491 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289090.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:38.947767 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289090.sst): second pass time = 1.52's, 71.363'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 42.342'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3506.115'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.13's, 277.205'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 30922.204'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.363'MB/sec zip pipeline throughput = 96.493'MB/sec entries = 930976 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.17 avg-zval = 32.20 usrkeys = 930976 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.17 avg-zval = 32.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930976 bytes = 3142064 } seq expand size = 6516832 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0300 GB dict = 1.08 MB all = 0.0360 GB } UnZip/Zip{ index = 10.5147 value = 3.6081 dict = 1.00 all = 3.7312 } Zip/UnZip{ index = 0.0951 value = 0.2772 dict = 1.00 all = 0.2680 } ---------------------------- total value len = 2252.702110 GB avg = 0.200 KB (by entry num) total key len = 307.195064 GB avg = 0.027 KB total ukey len = 237.577657 GB avg = 0.015 KB total ukey num = 15.749807663 Billion total entry num = 11.291121611 Billion write speed all = 40.081855612 MB/sec (with seq num) write speed all = 38.667520347 MB/sec (without seq num) 2024/11/24-02:39:38.990618 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289090: 930976 keys, 35971824 bytes, temperature: kWarm 2024/11/24-02:39:38.990697 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387178990660, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289090, "file_size": 35971824, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679891, "largest_seqno": 4399306597, "table_properties": {"data_size": 29974208, "index_size": 1770816, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26067328, "raw_average_key_size": 28, "raw_value_size": 108150532, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930976, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387175, "oldest_key_time": 0, "file_creation_time": 1732387175, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289090, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:39.038343 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289091.sst) /dev/shm/ToplingTemp/Topling-250155-vngrjK 2024/11/24-02:39:40.118085 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289091.sst): first pass time = 1.08's, 124.315'MB/sec 2024/11/24-02:39:41.685752 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289091.sst): old prealloc_size = 147639500, real_size = 31484528 2024/11/24-02:39:41.686348 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289091.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1085 GB zip-val = 0.0295 GB avg-val = 118.00 avg-zval = 32.07 2024/11/24-02:39:41.687103 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289091.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:42.513099 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289091.sst): second pass time = 1.57's, 69.250'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 42.846'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3179.133'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.331'MB/sec write SST data time = 0.13's, 268.182'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 35311.290'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 69.250'MB/sec zip pipeline throughput = 127.789'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 32.07 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 32.07 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3102656 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.09 MB all = 0.0354 GB } UnZip/Zip{ index = 10.3779 value = 3.6793 dict = 1.00 all = 3.7865 } Zip/UnZip{ index = 0.0964 value = 0.2718 dict = 1.00 all = 0.2641 } ---------------------------- total value len = 2252.817023 GB avg = 0.200 KB (by entry num) total key len = 307.220805 GB avg = 0.027 KB total ukey len = 237.596043 GB avg = 0.015 KB total ukey num = 15.750726963 Billion total entry num = 11.292040911 Billion write speed all = 40.081820356 MB/sec (with seq num) write speed all = 38.667448895 MB/sec (without seq num) 2024/11/24-02:39:42.575466 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289091: 919300 keys, 35446386 bytes, temperature: kWarm 2024/11/24-02:39:42.575514 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387182575486, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289091, "file_size": 35446386, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4285768131, "largest_seqno": 4395217468, "table_properties": {"data_size": 29482848, "index_size": 1771648, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387179, "oldest_key_time": 0, "file_creation_time": 1732387179, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289091, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:42.618094 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289092.sst) /dev/shm/ToplingTemp/Topling-250155-aehU5I 2024/11/24-02:39:43.619582 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289092.sst): first pass time = 1.00's, 134.022'MB/sec 2024/11/24-02:39:45.217744 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289092.sst): old prealloc_size = 147639500, real_size = 32904640 2024/11/24-02:39:45.218321 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289092.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930148 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.30 avg-zval = 33.22 2024/11/24-02:39:45.219077 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289092.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:46.094831 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289092.sst): second pass time = 1.60's, 67.718'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.067'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3183.752'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.316'MB/sec write SST data time = 0.14's, 262.752'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 28714.290'MB/sec zip my value time = 1.60's, unzip length = 0.108'GB zip my value throughput = 67.718'MB/sec zip pipeline throughput = 118.591'MB/sec entries = 930148 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.30 avg-zval = 33.22 usrkeys = 930148 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.30 avg-zval = 33.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930148 bytes = 3139264 } seq expand size = 6511036 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5133 value = 3.5005 dict = 1.00 all = 3.6378 } Zip/UnZip{ index = 0.0951 value = 0.2857 dict = 1.00 all = 0.2749 } ---------------------------- total value len = 2252.931708 GB avg = 0.199 KB (by entry num) total key len = 307.246849 GB avg = 0.027 KB total ukey len = 237.614646 GB avg = 0.015 KB total ukey num = 15.751657111 Billion total entry num = 11.292971059 Billion write speed all = 40.081775981 MB/sec (with seq num) write speed all = 38.667367334 MB/sec (without seq num) 2024/11/24-02:39:46.141060 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289092: 930148 keys, 36895712 bytes, temperature: kWarm 2024/11/24-02:39:46.141119 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387186141088, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289092, "file_size": 36895712, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679902, "largest_seqno": 4399306346, "table_properties": {"data_size": 30902448, "index_size": 1769472, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26044144, "raw_average_key_size": 28, "raw_value_size": 108173700, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930148, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387182, "oldest_key_time": 0, "file_creation_time": 1732387182, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289092, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:46.178050 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289093.sst) /dev/shm/ToplingTemp/Topling-250155-S6bXiL 2024/11/24-02:39:47.209193 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289093.sst): first pass time = 1.03's, 130.168'MB/sec 2024/11/24-02:39:48.526263 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289093.sst): old prealloc_size = 147639500, real_size = 30643456 2024/11/24-02:39:48.526834 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289093.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927409 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0286 GB avg-val = 116.72 avg-zval = 30.88 2024/11/24-02:39:48.527608 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289093.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:49.346682 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289093.sst): second pass time = 1.32's, 82.235'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 42.207'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3269.588'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.13's, 268.043'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 26631.255'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.235'MB/sec zip pipeline throughput = 82.235'MB/sec entries = 927409 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.72 avg-zval = 30.88 usrkeys = 927409 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.72 avg-zval = 30.88 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927409 bytes = 3130016 } seq expand size = 6491863 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0286 GB dict = 1.08 MB all = 0.0346 GB } UnZip/Zip{ index = 10.4449 value = 3.7803 dict = 1.00 all = 3.8762 } Zip/UnZip{ index = 0.0957 value = 0.2645 dict = 1.00 all = 0.2580 } ---------------------------- total value len = 2253.046450 GB avg = 0.199 KB (by entry num) total key len = 307.272816 GB avg = 0.027 KB total ukey len = 237.633194 GB avg = 0.015 KB total ukey num = 15.752584520 Billion total entry num = 11.293898468 Billion write speed all = 40.081938327 MB/sec (with seq num) write speed all = 38.667485535 MB/sec (without seq num) 2024/11/24-02:39:49.392654 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289093: 927409 keys, 34626046 bytes, temperature: kWarm 2024/11/24-02:39:49.392725 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387189392690, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289093, "file_size": 34626046, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4323454537, "largest_seqno": 4399306363, "table_properties": {"data_size": 28635632, "index_size": 1775808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25967452, "raw_average_key_size": 28, "raw_value_size": 108250302, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927409, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387186, "oldest_key_time": 0, "file_creation_time": 1732387186, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289093, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:49.433644 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289094.sst) /dev/shm/ToplingTemp/Topling-250155-XZFRZK 2024/11/24-02:39:50.348363 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289094.sst): first pass time = 0.91's, 146.738'MB/sec 2024/11/24-02:39:51.664733 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289094.sst): old prealloc_size = 147639500, real_size = 32341152 2024/11/24-02:39:51.665306 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289094.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922059 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0304 GB avg-val = 117.56 avg-zval = 32.92 2024/11/24-02:39:51.666052 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289094.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:39:52.490082 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289094.sst): second pass time = 1.32's, 82.395'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 43.997'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3190.007'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.331'MB/sec write SST data time = 0.13's, 283.968'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34895.719'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.395'MB/sec zip pipeline throughput = 82.395'MB/sec entries = 922059 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.56 avg-zval = 32.92 usrkeys = 922059 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.56 avg-zval = 32.92 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922059 bytes = 3111968 } seq expand size = 6454413 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.5139 value = 3.5709 dict = 1.00 all = 3.6963 } Zip/UnZip{ index = 0.0951 value = 0.2800 dict = 1.00 all = 0.2705 } ---------------------------- total value len = 2253.161304 GB avg = 0.199 KB (by entry num) total key len = 307.298634 GB avg = 0.027 KB total ukey len = 237.651636 GB avg = 0.015 KB total ukey num = 15.753506579 Billion total entry num = 11.294820527 Billion write speed all = 40.082168110 MB/sec (with seq num) write speed all = 38.667669447 MB/sec (without seq num) 2024/11/24-02:39:52.535995 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289094: 922059 keys, 36311630 bytes, temperature: kWarm 2024/11/24-02:39:52.536065 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387192536030, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289094, "file_size": 36311630, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4265679921, "largest_seqno": 4399305198, "table_properties": {"data_size": 30356496, "index_size": 1753984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25817652, "raw_average_key_size": 28, "raw_value_size": 108400106, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922059, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387189, "oldest_key_time": 0, "file_creation_time": 1732387189, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289094, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:52.577347 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289095.sst) /dev/shm/ToplingTemp/Topling-250155-EwL4FK 2024/11/24-02:39:52.840748 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289095.sst): first pass time = 0.26's, 241.546'MB/sec 2024/11/24-02:39:53.471865 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289095.sst): old prealloc_size = 147639500, real_size = 16178016 2024/11/24-02:39:53.472206 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289095.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 439253 prefix = 4 raw-key = 0.0070 GB zip-key = 0.0008 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0513 GB zip-val = 0.0152 GB avg-val = 116.82 avg-zval = 34.66 2024/11/24-02:39:53.472572 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289095.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0010 GB, waited 0.000 sec, Key+Value = 0.064 GB 2024/11/24-02:39:53.894280 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00/289095.sst): second pass time = 0.63's, 81.411'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.42's, 42.656'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2655.156'MB/sec (index lex order gen) rebuild zvType time = 0.36's, 0.306'MB/sec write SST data time = 0.06's, 284.422'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.519'MB, throughput = 16216.856'MB/sec zip my value time = 0.63's, unzip length = 0.051'GB zip my value throughput = 81.411'MB/sec zip pipeline throughput = 81.411'MB/sec entries = 439253 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.82 avg-zval = 34.66 usrkeys = 439253 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.82 avg-zval = 34.66 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 439253 bytes = 1427584 } seq expand size = 3074771 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0088 GB value = 0.0513 GB dict = 0.52 MB all = 0.0636 GB } __ZipSize{ index = 0.0008 GB value = 0.0152 GB dict = 0.52 MB all = 0.0180 GB } UnZip/Zip{ index = 10.4322 value = 3.3703 dict = 1.00 all = 3.5307 } Zip/UnZip{ index = 0.0959 value = 0.2967 dict = 1.00 all = 0.2832 } ---------------------------- total value len = 2253.215695 GB avg = 0.199 KB (by entry num) total key len = 307.310933 GB avg = 0.027 KB total ukey len = 237.660421 GB avg = 0.015 KB total ukey num = 15.753945832 Billion total entry num = 11.295259780 Billion write speed all = 40.082331008 MB/sec (with seq num) write speed all = 38.667808429 MB/sec (without seq num) 2024/11/24-02:39:53.921767 139957643191872 [compaction_job.cc:1948] [default] [JOB 40235] Generated table #289095: 439253 keys, 18017453 bytes, temperature: kWarm 2024/11/24-02:39:53.921836 139957643191872 EVENT_LOG_v1 {"time_micros": 1732387193921801, "cf_name": "default", "job": 40235, "event": "table_file_creation", "file_number": 289095, "file_size": 18017453, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4355082273, "largest_seqno": 4399306374, "table_properties": {"data_size": 15225936, "index_size": 842112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12299084, "raw_average_key_size": 28, "raw_value_size": 51315722, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 439253, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732387192, "oldest_key_time": 0, "file_creation_time": 1732387192, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289095, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:39:53.939285 139957643191872 [compaction_job.cc:703] job-40235: subcompact[0], size: 0.592508 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40235/att-00 [289079,289080,289081,289082,289083,289084,289085,289086,289087,289088,289089,289090,289091,289092,289093,289094,289095] 2024/11/24-02:39:54.038529 139957643191872 (Original Log Time 2024/11/24-02:38:58.060403) EVENT_LOG_v1 {"time_micros": 1732387138060355, "job": 40235, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [289034], "files_L3": [289069, 288993, 288994, 288995, 288996, 288997, 288998, 288999, 289000, 289001, 289002, 289003, 289004, 289005, 288962, 288963, 288964, 289070], "score": 0.613993, "input_data_size": 613119003, "oldest_snapshot_seqno": 4399487091} 2024/11/24-02:39:54.038533 139957643191872 (Original Log Time 2024/11/24-02:39:54.037125) [compaction_job.cc:2022] [default] [JOB 40235] Compacted 1@2 + 18@3 files to L3 => 592508455 bytes 2024/11/24-02:39:54.038535 139957643191872 (Original Log Time 2024/11/24-02:39:54.038464) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 11.0 rd, 10.6 wr, level 3, files in(1, 18) out(17 +0 blob) MB in(27.1, 557.6 +0.0 blob) out(565.1 +0.0 blob), read-write-amplify(42.4) write-amplify(20.9) OK, records in: 15384647, records dropped: 99978 output_compression: Snappy 2024/11/24-02:39:54.038540 139957643191872 (Original Log Time 2024/11/24-02:39:54.038508) EVENT_LOG_v1 {"time_micros": 1732387194038487, "cf": "default", "job": 40235, "event": "compaction_finished", "compaction_time_micros": 55878890, "compaction_time_cpu_micros": 24325857, "output_level": 3, "num_output_files": 17, "total_output_size": 592508455, "num_input_records": 15384647, "num_output_records": 15284669, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 407402370, "file_range_sync_nanos": 0, "file_fsync_nanos": 11694, "file_prepare_write_nanos": 45301, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/24-02:39:54.177607 139957643191872 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732387137,"executesMs":56122,"compactionJobId":40235,"attempt":0,"compactionInputRawBytes":2225296558,"compactionInputZipBytes":613119003,"compactionOutputRawBytes":0,"compactionOutputZipBytes":592508455,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-fb83e64b967b937276fa28efc1c9ab06-fa43ac2fcfbdb02c-00"}}, response = 2024/11/24-02:39:54.177698 139957643191872 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40235/att-00: olev 3, work 56.122 s, result 7.725 ms, install 1.432 ms, input{raw 2.072 GiB zip 584.716 MiB}