2024/11/23-19:08:37.861323 139957559264832 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00: fork to child time = 0.021597 sec 2024/11/23-19:08:37.862342 139957559264832 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00 2024/11/23-19:08:37.872098 139957559264832 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00 2024/11/23-19:08:37.873288 139957559264832 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-19:08:37.873640 139957559264832 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-19:08:37.929295 139957559264832 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-33865/att-00/MANIFEST-206347 2024/11/23-19:08:37.944952 139957559264832 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-33865/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 276485, last_sequence is 4205813831, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 276324 2024/11/23-19:08:37.944961 139957559264832 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-19:08:37.945134 139957559264832 [version_set.cc:5721] Creating manifest 276485 2024/11/23-19:08:37.978598 139957559264832 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-19:08:38.001826 139957559264832 [compaction_job.cc:2437] [default] [JOB 33865] Compacting 1@2 + 16@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/23-19:08:38.001850 139957559264832 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 25357 Base level 2, inputs: [276457(27MB)], [276404(37MB) 276405(37MB) 276406(37MB) 276407(38MB) 276408(38MB) 276409(35MB) 276410(36MB) 276411(33MB) 276412(35MB) 276413(36MB) 276414(35MB) 276415(35MB) 276416(35MB) 276417(36MB) 276418(37MB) 276478(37MB)] 2024/11/23-19:08:38.010911 139957559264832 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-19:08:38.010958 139957559264832 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-19:08:38.010993 139957559264832 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-19:08:38.011003 139957559264832 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-19:08:38.011018 139957559264832 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-19:08:38.668648 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276486.sst) /dev/shm/ToplingTemp/Topling-161098-7w9n7F 2024/11/23-19:08:39.556944 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276486.sst): first pass time = 0.89's, 151.105'MB/sec 2024/11/23-19:08:42.508294 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276486.sst): old prealloc_size = 147639500, real_size = 35448336 2024/11/23-19:08:42.508921 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276486.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928742 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0334 GB avg-val = 116.52 avg-zval = 36.02 2024/11/23-19:08:42.509832 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276486.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:08:43.486395 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276486.sst): second pass time = 2.95's, 36.676'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 40.320'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2883.882'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.313'MB/sec write SST data time = 0.24's, 167.291'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35853.623'MB/sec zip my value time = 2.95's, unzip length = 0.108'GB zip my value throughput = 36.676'MB/sec zip pipeline throughput = 113.295'MB/sec entries = 928742 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.52 avg-zval = 36.02 usrkeys = 928742 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.52 avg-zval = 36.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928742 bytes = 3134512 } seq expand size = 6501194 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0334 GB dict = 1.08 MB all = 0.0394 GB } UnZip/Zip{ index = 10.5130 value = 3.2352 dict = 1.00 all = 3.4034 } Zip/UnZip{ index = 0.0951 value = 0.3091 dict = 1.00 all = 0.2938 } ---------------------------- total value len = 8.963334 GB avg = 0.168 KB (by entry num) total key len = 1.466826 GB avg = 0.028 KB total ukey len = 1.016453 GB avg = 0.015 KB total ukey num = 0.066479177 Billion total entry num = 0.053286906 Billion write speed all = 0.283341471 MB/sec (with seq num) write speed all = 0.271760908 MB/sec (without seq num) 2024/11/23-19:08:43.539874 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276486: 928742 keys, 39436128 bytes, temperature: kWarm 2024/11/23-19:08:43.540017 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360123539921, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276486, "file_size": 39436128, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4136224572, "largest_seqno": 4205696529, "table_properties": {"data_size": 33449088, "index_size": 1766848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26004776, "raw_average_key_size": 28, "raw_value_size": 108213040, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928742, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360118, "oldest_key_time": 0, "file_creation_time": 1732360118, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276486, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:08:43.613338 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276487.sst) /dev/shm/ToplingTemp/Topling-161098-79YBmH 2024/11/23-19:08:45.057323 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276487.sst): first pass time = 1.44's, 92.953'MB/sec 2024/11/23-19:08:47.485970 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276487.sst): old prealloc_size = 147639500, real_size = 35658048 2024/11/23-19:08:47.486623 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276487.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929097 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0336 GB avg-val = 116.46 avg-zval = 36.21 2024/11/23-19:08:47.487418 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276487.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:08:48.626987 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276487.sst): second pass time = 2.43's, 44.571'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.14's, 34.646'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2894.935'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.255'MB/sec write SST data time = 0.23's, 173.494'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 28700.476'MB/sec zip my value time = 2.43's, unzip length = 0.108'GB zip my value throughput = 44.571'MB/sec zip pipeline throughput = 100.899'MB/sec entries = 929097 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.46 avg-zval = 36.21 usrkeys = 929097 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.46 avg-zval = 36.21 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929097 bytes = 3019584 } seq expand size = 6503679 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0395 GB } UnZip/Zip{ index = 10.4429 value = 3.2159 dict = 1.00 all = 3.3953 } Zip/UnZip{ index = 0.0958 value = 0.3110 dict = 1.00 all = 0.2945 } ---------------------------- total value len = 9.078041 GB avg = 0.167 KB (by entry num) total key len = 1.492841 GB avg = 0.028 KB total ukey len = 1.035035 GB avg = 0.015 KB total ukey num = 0.067408274 Billion total entry num = 0.054216003 Billion write speed all = 0.287124156 MB/sec (with seq num) write speed all = 0.275343323 MB/sec (without seq num) 2024/11/23-19:08:48.675831 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276487: 929097 keys, 39530440 bytes, temperature: kWarm 2024/11/23-19:08:48.675940 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360128675874, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276487, "file_size": 39530440, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4148263343, "largest_seqno": 4205696424, "table_properties": {"data_size": 33646192, "index_size": 1779392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26014716, "raw_average_key_size": 28, "raw_value_size": 108203038, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929097, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360123, "oldest_key_time": 0, "file_creation_time": 1732360123, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276487, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:08:48.731552 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276488.sst) /dev/shm/ToplingTemp/Topling-161098-N9wimE 2024/11/23-19:08:49.667560 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276488.sst): first pass time = 0.94's, 143.399'MB/sec 2024/11/23-19:08:51.437635 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276488.sst): old prealloc_size = 147639500, real_size = 33668784 2024/11/23-19:08:51.438227 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276488.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926108 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0317 GB avg-val = 116.93 avg-zval = 34.19 2024/11/23-19:08:51.439011 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276488.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:08:52.477698 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276488.sst): second pass time = 1.77's, 61.202'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 36.315'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3107.298'MB/sec (index lex order gen) rebuild zvType time = 0.91's, 0.255'MB/sec write SST data time = 0.13's, 286.945'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 38450.628'MB/sec zip my value time = 1.77's, unzip length = 0.108'GB zip my value throughput = 61.202'MB/sec zip pipeline throughput = 117.099'MB/sec entries = 926108 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.93 avg-zval = 34.19 usrkeys = 926108 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.93 avg-zval = 34.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926108 bytes = 3241392 } seq expand size = 6482756 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.4434 value = 3.4199 dict = 1.00 all = 3.5537 } Zip/UnZip{ index = 0.0958 value = 0.2924 dict = 1.00 all = 0.2814 } ---------------------------- total value len = 9.192810 GB avg = 0.167 KB (by entry num) total key len = 1.518772 GB avg = 0.028 KB total ukey len = 1.053557 GB avg = 0.015 KB total ukey num = 0.068334382 Billion total entry num = 0.055142111 Billion write speed all = 0.290915410 MB/sec (with seq num) write speed all = 0.278934592 MB/sec (without seq num) 2024/11/23-19:08:52.569386 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276488: 926108 keys, 37768658 bytes, temperature: kWarm 2024/11/23-19:08:52.569455 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360132569418, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276488, "file_size": 37768658, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4003482220, "largest_seqno": 4205696108, "table_properties": {"data_size": 31663520, "index_size": 1773568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25931024, "raw_average_key_size": 28, "raw_value_size": 108286828, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926108, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360128, "oldest_key_time": 0, "file_creation_time": 1732360128, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276488, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:08:52.619538 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276489.sst) /dev/shm/ToplingTemp/Topling-161098-c16ySE 2024/11/23-19:08:53.609684 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276489.sst): first pass time = 0.99's, 135.558'MB/sec 2024/11/23-19:08:55.593739 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276489.sst): old prealloc_size = 147639500, real_size = 36537168 2024/11/23-19:08:55.594433 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276489.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933945 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0345 GB avg-val = 115.71 avg-zval = 36.97 2024/11/23-19:08:55.595215 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276489.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:08:56.569436 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276489.sst): second pass time = 1.98's, 54.510'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 41.131'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2645.061'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.287'MB/sec write SST data time = 0.16's, 250.593'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34433.983'MB/sec zip my value time = 1.98's, unzip length = 0.108'GB zip my value throughput = 54.510'MB/sec zip pipeline throughput = 109.126'MB/sec entries = 933945 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.71 avg-zval = 36.97 usrkeys = 933945 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.07 avg-zval = 36.97 TagRS{ kinds = RS_Key0_TagN bytes = 146040 } TagArray{ size = 715336 bytes = 2593104 } seq expand size = 5007352 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.5136 value = 3.1299 dict = 1.00 all = 3.3445 } Zip/UnZip{ index = 0.0951 value = 0.3195 dict = 1.00 all = 0.2990 } ---------------------------- total value len = 9.305885 GB avg = 0.166 KB (by entry num) total key len = 1.544922 GB avg = 0.028 KB total ukey len = 1.072236 GB avg = 0.015 KB total ukey num = 0.069268327 Billion total entry num = 0.056076056 Billion write speed all = 0.294663874 MB/sec (with seq num) write speed all = 0.282481489 MB/sec (without seq num) 2024/11/23-19:08:56.641195 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276489: 933945 keys, 40130560 bytes, temperature: kWarm 2024/11/23-19:08:56.641254 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360136641220, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276489, "file_size": 40130560, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 4012776165, "table_properties": {"data_size": 34526848, "index_size": 1776640, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26150460, "raw_average_key_size": 28, "raw_value_size": 108067298, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933945, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360132, "oldest_key_time": 0, "file_creation_time": 1732360132, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276489, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:08:56.691173 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276490.sst) /dev/shm/ToplingTemp/Topling-161098-MEmUaH 2024/11/23-19:08:57.901143 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276490.sst): first pass time = 1.21's, 110.930'MB/sec 2024/11/23-19:08:59.635479 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276490.sst): old prealloc_size = 147639500, real_size = 35606000 2024/11/23-19:08:59.636070 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276490.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929788 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0336 GB avg-val = 116.35 avg-zval = 36.12 2024/11/23-19:08:59.636833 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276490.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:00.477376 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276490.sst): second pass time = 1.73's, 62.402'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 47.231'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3146.769'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.13's, 297.803'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 32300.748'MB/sec zip my value time = 1.73's, unzip length = 0.108'GB zip my value throughput = 62.402'MB/sec zip pipeline throughput = 75.647'MB/sec entries = 929788 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.35 avg-zval = 36.12 usrkeys = 929788 avg-key = 20.00 avg-zkey = 1.93 avg-val = 122.69 avg-zval = 36.12 TagRS{ kinds = RS_Key0_TagN bytes = 145360 } TagArray{ size = 841605 bytes = 3156032 } seq expand size = 5891235 multi value expand size = 0 cnt WriteAppend = 20 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0336 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.3749 value = 3.2216 dict = 1.00 all = 3.3754 } Zip/UnZip{ index = 0.0964 value = 0.3104 dict = 1.00 all = 0.2963 } ---------------------------- total value len = 9.419960 GB avg = 0.165 KB (by entry num) total key len = 1.570956 GB avg = 0.028 KB total ukey len = 1.090832 GB avg = 0.016 KB total ukey num = 0.070198115 Billion total entry num = 0.057005844 Billion write speed all = 0.298436997 MB/sec (with seq num) write speed all = 0.286053933 MB/sec (without seq num) 2024/11/23-19:09:00.526583 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276490: 929788 keys, 39763600 bytes, temperature: kWarm 2024/11/23-19:09:00.526646 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360140526612, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276490, "file_size": 39763600, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 4012775967, "table_properties": {"data_size": 33580960, "index_size": 1792384, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26034064, "raw_average_key_size": 28, "raw_value_size": 108183804, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929788, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360136, "oldest_key_time": 0, "file_creation_time": 1732360136, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276490, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:00.563956 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276491.sst) /dev/shm/ToplingTemp/Topling-161098-cmLe2E 2024/11/23-19:09:01.801936 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276491.sst): first pass time = 1.24's, 108.422'MB/sec 2024/11/23-19:09:03.621621 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276491.sst): old prealloc_size = 147639500, real_size = 32739824 2024/11/23-19:09:03.622341 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276491.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928520 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0307 GB avg-val = 116.55 avg-zval = 33.11 2024/11/23-19:09:03.623249 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276491.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:04.518117 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276491.sst): second pass time = 1.82's, 59.502'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 41.357'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2526.133'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.316'MB/sec write SST data time = 0.16's, 229.100'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34596.386'MB/sec zip my value time = 1.82's, unzip length = 0.108'GB zip my value throughput = 59.502'MB/sec zip pipeline throughput = 105.462'MB/sec entries = 928520 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.55 avg-zval = 33.11 usrkeys = 928520 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.55 avg-zval = 33.11 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928520 bytes = 3481968 } seq expand size = 6499640 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5150 value = 3.5203 dict = 1.00 all = 3.6202 } Zip/UnZip{ index = 0.0951 value = 0.2841 dict = 1.00 all = 0.2762 } ---------------------------- total value len = 9.534679 GB avg = 0.165 KB (by entry num) total key len = 1.596955 GB avg = 0.028 KB total ukey len = 1.109402 GB avg = 0.016 KB total ukey num = 0.071126635 Billion total entry num = 0.057934364 Billion write speed all = 0.302224748 MB/sec (with seq num) write speed all = 0.289641367 MB/sec (without seq num) 2024/11/23-19:09:04.572076 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276491: 928520 keys, 37075156 bytes, temperature: kWarm 2024/11/23-19:09:04.572140 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360144572107, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276491, "file_size": 37075156, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 4012776021, "table_properties": {"data_size": 30741408, "index_size": 1766080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25998560, "raw_average_key_size": 28, "raw_value_size": 108219308, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928520, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360140, "oldest_key_time": 0, "file_creation_time": 1732360140, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276491, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:04.611008 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276492.sst) /dev/shm/ToplingTemp/Topling-161098-tyr6kH 2024/11/23-19:09:05.741078 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276492.sst): first pass time = 1.13's, 118.773'MB/sec 2024/11/23-19:09:07.756711 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276492.sst): old prealloc_size = 147639500, real_size = 35304384 2024/11/23-19:09:07.757324 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276492.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925052 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0333 GB avg-val = 117.09 avg-zval = 36.00 2024/11/23-19:09:07.758027 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276492.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:08.580383 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276492.sst): second pass time = 2.01's, 53.757'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 47.461'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2984.721'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 299.549'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 22666.373'MB/sec zip my value time = 2.01's, unzip length = 0.108'GB zip my value throughput = 53.757'MB/sec zip pipeline throughput = 95.481'MB/sec entries = 925052 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.09 avg-zval = 36.00 usrkeys = 925052 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.71 avg-zval = 36.00 TagRS{ kinds = RS_Key0_TagN bytes = 144640 } TagArray{ size = 742055 bytes = 2782720 } seq expand size = 5194385 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0333 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.4440 value = 3.2526 dict = 1.00 all = 3.4335 } Zip/UnZip{ index = 0.0957 value = 0.3074 dict = 1.00 all = 0.2913 } ---------------------------- total value len = 9.648190 GB avg = 0.164 KB (by entry num) total key len = 1.622856 GB avg = 0.028 KB total ukey len = 1.127903 GB avg = 0.016 KB total ukey num = 0.072051687 Billion total entry num = 0.058859416 Billion write speed all = 0.305976055 MB/sec (with seq num) write speed all = 0.293193162 MB/sec (without seq num) 2024/11/23-19:09:08.630260 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276492: 925052 keys, 39091186 bytes, temperature: kWarm 2024/11/23-19:09:08.630327 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360148630293, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276492, "file_size": 39091186, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 4012776210, "table_properties": {"data_size": 33301488, "index_size": 1771456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25901456, "raw_average_key_size": 28, "raw_value_size": 108316372, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925052, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360144, "oldest_key_time": 0, "file_creation_time": 1732360144, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276492, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:08.669462 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276493.sst) /dev/shm/ToplingTemp/Topling-161098-d54lTE 2024/11/23-19:09:09.733489 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276493.sst): first pass time = 1.06's, 126.146'MB/sec 2024/11/23-19:09:11.823568 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276493.sst): old prealloc_size = 147639500, real_size = 32091216 2024/11/23-19:09:11.824215 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276493.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933457 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0301 GB avg-val = 115.79 avg-zval = 32.23 2024/11/23-19:09:11.825057 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276493.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:12.849968 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276493.sst): second pass time = 2.09's, 51.732'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.03's, 35.393'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2843.331'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.297'MB/sec write SST data time = 0.24's, 151.581'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 28611.290'MB/sec zip my value time = 2.09's, unzip length = 0.108'GB zip my value throughput = 51.732'MB/sec zip pipeline throughput = 161.881'MB/sec entries = 933457 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.79 avg-zval = 32.23 usrkeys = 933457 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.79 avg-zval = 32.23 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933457 bytes = 3383792 } seq expand size = 6534199 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.5153 value = 3.5929 dict = 1.00 all = 3.6949 } Zip/UnZip{ index = 0.0951 value = 0.2783 dict = 1.00 all = 0.2706 } ---------------------------- total value len = 9.762805 GB avg = 0.163 KB (by entry num) total key len = 1.648993 GB avg = 0.028 KB total ukey len = 1.146572 GB avg = 0.016 KB total ukey num = 0.072985144 Billion total entry num = 0.059792873 Billion write speed all = 0.309761159 MB/sec (with seq num) write speed all = 0.296777046 MB/sec (without seq num) 2024/11/23-19:09:12.904199 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276493: 933457 keys, 36325606 bytes, temperature: kWarm 2024/11/23-19:09:12.904292 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360152904242, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276493, "file_size": 36325606, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3480997236, "largest_seqno": 4012776228, "table_properties": {"data_size": 30082240, "index_size": 1775424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26136796, "raw_average_key_size": 28, "raw_value_size": 108081050, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933457, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360148, "oldest_key_time": 0, "file_creation_time": 1732360148, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276493, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:12.973176 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276494.sst) /dev/shm/ToplingTemp/Topling-161098-fvtkVD 2024/11/23-19:09:14.011470 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276494.sst): first pass time = 1.04's, 129.275'MB/sec 2024/11/23-19:09:15.507467 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276494.sst): old prealloc_size = 147639500, real_size = 33063040 2024/11/23-19:09:15.508034 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276494.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 933010 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0310 GB avg-val = 115.85 avg-zval = 33.27 2024/11/23-19:09:15.508803 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276494.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:16.572410 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276494.sst): second pass time = 1.50's, 72.291'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 35.022'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3270.350'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.267'MB/sec write SST data time = 0.19's, 194.331'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 37002.335'MB/sec zip my value time = 1.50's, unzip length = 0.108'GB zip my value throughput = 72.291'MB/sec zip pipeline throughput = 84.804'MB/sec entries = 933010 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.85 avg-zval = 33.27 usrkeys = 933010 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.85 avg-zval = 33.27 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 933010 bytes = 3382176 } seq expand size = 6531070 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0187 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.4455 value = 3.4820 dict = 1.00 all = 3.5988 } Zip/UnZip{ index = 0.0957 value = 0.2872 dict = 1.00 all = 0.2779 } ---------------------------- total value len = 9.877430 GB avg = 0.163 KB (by entry num) total key len = 1.675117 GB avg = 0.028 KB total ukey len = 1.165232 GB avg = 0.016 KB total ukey num = 0.073918154 Billion total entry num = 0.060725883 Billion write speed all = 0.313549958 MB/sec (with seq num) write speed all = 0.300364573 MB/sec (without seq num) 2024/11/23-19:09:16.619487 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276494: 933010 keys, 37295326 bytes, temperature: kWarm 2024/11/23-19:09:16.619546 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360156619514, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276494, "file_size": 37295326, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3512539342, "largest_seqno": 4012776248, "table_properties": {"data_size": 31043184, "index_size": 1786432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26124280, "raw_average_key_size": 28, "raw_value_size": 108093584, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 933010, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360152, "oldest_key_time": 0, "file_creation_time": 1732360152, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276494, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:16.660291 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276495.sst) /dev/shm/ToplingTemp/Topling-161098-mkBKdG 2024/11/23-19:09:17.581901 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276495.sst): first pass time = 0.92's, 145.639'MB/sec 2024/11/23-19:09:19.134338 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276495.sst): old prealloc_size = 147639500, real_size = 34872656 2024/11/23-19:09:19.134933 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276495.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925568 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0329 GB avg-val = 117.01 avg-zval = 35.51 2024/11/23-19:09:19.136681 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276495.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:19.980451 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276495.sst): second pass time = 1.55's, 69.797'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 46.329'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3102.577'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.13's, 308.011'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32298.003'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 69.797'MB/sec zip pipeline throughput = 71.568'MB/sec entries = 925568 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.01 avg-zval = 35.51 usrkeys = 925568 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.01 avg-zval = 35.51 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925568 bytes = 3470896 } seq expand size = 6478976 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.4430 value = 3.2950 dict = 1.00 all = 3.4241 } Zip/UnZip{ index = 0.0958 value = 0.3035 dict = 1.00 all = 0.2921 } ---------------------------- total value len = 9.992210 GB avg = 0.162 KB (by entry num) total key len = 1.701033 GB avg = 0.028 KB total ukey len = 1.183744 GB avg = 0.016 KB total ukey num = 0.074843722 Billion total entry num = 0.061651451 Billion write speed all = 0.317339282 MB/sec (with seq num) write speed all = 0.303954166 MB/sec (without seq num) 2024/11/23-19:09:20.029200 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276495: 925568 keys, 39198466 bytes, temperature: kWarm 2024/11/23-19:09:20.029269 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360160029233, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276495, "file_size": 39198466, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3411274482, "largest_seqno": 4012776268, "table_properties": {"data_size": 32868480, "index_size": 1772608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25915904, "raw_average_key_size": 28, "raw_value_size": 108301824, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925568, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360156, "oldest_key_time": 0, "file_creation_time": 1732360156, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276495, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:20.064356 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276496.sst) /dev/shm/ToplingTemp/Topling-161098-0GPBHG 2024/11/23-19:09:20.985639 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276496.sst): first pass time = 0.92's, 145.692'MB/sec 2024/11/23-19:09:22.342360 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276496.sst): old prealloc_size = 147639500, real_size = 32687952 2024/11/23-19:09:22.342944 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276496.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931242 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1081 GB zip-val = 0.0307 GB avg-val = 116.13 avg-zval = 32.95 2024/11/23-19:09:22.343797 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276496.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:23.198566 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276496.sst): second pass time = 1.36's, 79.776'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.116'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3159.547'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.320'MB/sec write SST data time = 0.13's, 288.129'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 25324.693'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.776'MB/sec zip pipeline throughput = 79.776'MB/sec entries = 931242 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.13 avg-zval = 32.95 usrkeys = 931242 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.13 avg-zval = 32.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931242 bytes = 3375760 } seq expand size = 6518694 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0369 GB } UnZip/Zip{ index = 10.5120 value = 3.5245 dict = 1.00 all = 3.6358 } Zip/UnZip{ index = 0.0951 value = 0.2837 dict = 1.00 all = 0.2750 } ---------------------------- total value len = 10.106872 GB avg = 0.161 KB (by entry num) total key len = 1.727108 GB avg = 0.028 KB total ukey len = 1.202369 GB avg = 0.016 KB total ukey num = 0.075774964 Billion total entry num = 0.062582693 Billion write speed all = 0.321130638 MB/sec (with seq num) write speed all = 0.307544528 MB/sec (without seq num) 2024/11/23-19:09:23.245149 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276496: 931242 keys, 36915356 bytes, temperature: kWarm 2024/11/23-19:09:23.245216 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360163245181, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276496, "file_size": 36915356, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3525889795, "largest_seqno": 4012776300, "table_properties": {"data_size": 30683200, "index_size": 1771776, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26074776, "raw_average_key_size": 28, "raw_value_size": 108142972, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931242, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360160, "oldest_key_time": 0, "file_creation_time": 1732360160, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276496, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:23.280393 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276497.sst) /dev/shm/ToplingTemp/Topling-161098-klSDPG 2024/11/23-19:09:24.174690 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276497.sst): first pass time = 0.89's, 150.088'MB/sec 2024/11/23-19:09:25.494193 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276497.sst): old prealloc_size = 147639500, real_size = 34591760 2024/11/23-19:09:25.494720 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276497.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922791 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0326 GB avg-val = 117.45 avg-zval = 35.33 2024/11/23-19:09:25.495536 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276497.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:26.338028 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276497.sst): second pass time = 1.32's, 82.186'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.113'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3487.636'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 309.781'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34995.540'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.186'MB/sec zip pipeline throughput = 82.186'MB/sec entries = 922791 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.45 avg-zval = 35.33 usrkeys = 922791 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.45 avg-zval = 35.33 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922791 bytes = 3460480 } seq expand size = 6459537 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0389 GB } UnZip/Zip{ index = 10.5134 value = 3.3240 dict = 1.00 all = 3.4494 } Zip/UnZip{ index = 0.0951 value = 0.3008 dict = 1.00 all = 0.2899 } ---------------------------- total value len = 10.221711 GB avg = 0.161 KB (by entry num) total key len = 1.752946 GB avg = 0.028 KB total ukey len = 1.220824 GB avg = 0.016 KB total ukey num = 0.076697755 Billion total entry num = 0.063505484 Billion write speed all = 0.324920423 MB/sec (with seq num) write speed all = 0.311135157 MB/sec (without seq num) 2024/11/23-19:09:26.384960 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276497: 922791 keys, 38910170 bytes, temperature: kWarm 2024/11/23-19:09:26.385042 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360166385007, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276497, "file_size": 38910170, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3405268578, "largest_seqno": 4012776344, "table_properties": {"data_size": 32605440, "index_size": 1755456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25838148, "raw_average_key_size": 28, "raw_value_size": 108379670, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922791, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360163, "oldest_key_time": 0, "file_creation_time": 1732360163, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276497, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:26.428493 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276498.sst) /dev/shm/ToplingTemp/Topling-161098-vpooCD 2024/11/23-19:09:27.290146 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276498.sst): first pass time = 0.86's, 155.775'MB/sec 2024/11/23-19:09:28.581275 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276498.sst): old prealloc_size = 147639500, real_size = 32923072 2024/11/23-19:09:28.581785 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276498.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930989 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0309 GB avg-val = 116.17 avg-zval = 33.20 2024/11/23-19:09:28.584003 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276498.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:29.442134 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276498.sst): second pass time = 1.29's, 83.815'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.290'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3658.283'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 294.993'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33918.082'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 83.815'MB/sec zip pipeline throughput = 83.815'MB/sec entries = 930989 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.17 avg-zval = 33.20 usrkeys = 930989 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.17 avg-zval = 33.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930989 bytes = 3491216 } seq expand size = 6516923 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.4435 value = 3.4992 dict = 1.00 all = 3.6017 } Zip/UnZip{ index = 0.0958 value = 0.2858 dict = 1.00 all = 0.2777 } ---------------------------- total value len = 10.336378 GB avg = 0.160 KB (by entry num) total key len = 1.779014 GB avg = 0.028 KB total ukey len = 1.239444 GB avg = 0.016 KB total ukey num = 0.077628744 Billion total entry num = 0.064436473 Billion write speed all = 0.328711434 MB/sec (with seq num) write speed all = 0.314725255 MB/sec (without seq num) 2024/11/23-19:09:29.488133 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276498: 930989 keys, 37265596 bytes, temperature: kWarm 2024/11/23-19:09:29.488205 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360169488169, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276498, "file_size": 37265596, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3471845708, "largest_seqno": 4012771944, "table_properties": {"data_size": 30907248, "index_size": 1782912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26067692, "raw_average_key_size": 28, "raw_value_size": 108150134, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930989, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360166, "oldest_key_time": 0, "file_creation_time": 1732360166, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276498, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:29.522662 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276499.sst) /dev/shm/ToplingTemp/Topling-161098-FIKC9G 2024/11/23-19:09:30.540235 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276499.sst): first pass time = 1.02's, 131.905'MB/sec 2024/11/23-19:09:31.939013 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276499.sst): old prealloc_size = 147639500, real_size = 34840576 2024/11/23-19:09:31.939574 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276499.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928678 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1082 GB zip-val = 0.0328 GB avg-val = 116.53 avg-zval = 35.35 2024/11/23-19:09:31.940069 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276499.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:32.715412 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276499.sst): second pass time = 1.40's, 77.410'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 48.364'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3356.198'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.357'MB/sec write SST data time = 0.12's, 301.162'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33394.187'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.410'MB/sec zip pipeline throughput = 77.410'MB/sec entries = 928678 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.53 avg-zval = 35.35 usrkeys = 928678 avg-key = 20.00 avg-zkey = 1.92 avg-val = 120.20 avg-zval = 35.35 TagRS{ kinds = RS_Key0_TagN bytes = 145200 } TagArray{ size = 487119 bytes = 1704928 } seq expand size = 3409833 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0328 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.4427 value = 3.2963 dict = 1.00 all = 3.5745 } Zip/UnZip{ index = 0.0958 value = 0.3034 dict = 1.00 all = 0.2798 } ---------------------------- total value len = 10.448003 GB avg = 0.160 KB (by entry num) total key len = 1.805017 GB avg = 0.028 KB total ukey len = 1.258018 GB avg = 0.016 KB total ukey num = 0.078557422 Billion total entry num = 0.065365151 Billion write speed all = 0.332415988 MB/sec (with seq num) write speed all = 0.318229496 MB/sec (without seq num) 2024/11/23-19:09:32.760851 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276499: 928678 keys, 37549039 bytes, temperature: kWarm 2024/11/23-19:09:32.760928 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360172760885, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276499, "file_size": 37549039, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 4012776360, "table_properties": {"data_size": 32829616, "index_size": 1778624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26002984, "raw_average_key_size": 28, "raw_value_size": 108214888, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928678, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360169, "oldest_key_time": 0, "file_creation_time": 1732360169, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276499, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:32.798188 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276500.sst) /dev/shm/ToplingTemp/Topling-161098-pSH7ED 2024/11/23-19:09:33.805316 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276500.sst): first pass time = 1.01's, 133.273'MB/sec 2024/11/23-19:09:35.291725 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276500.sst): old prealloc_size = 147639500, real_size = 36865536 2024/11/23-19:09:35.292325 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276500.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926692 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0348 GB avg-val = 116.84 avg-zval = 37.60 2024/11/23-19:09:35.292551 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276500.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:36.002106 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276500.sst): second pass time = 1.49's, 72.880'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.71's, 53.989'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3069.433'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.395'MB/sec write SST data time = 0.12's, 311.960'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30753.282'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.880'MB/sec zip pipeline throughput = 72.880'MB/sec entries = 926692 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.84 avg-zval = 37.60 usrkeys = 926692 avg-key = 20.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 37.60 TagRS{ kinds = RS_Key0_TagN bytes = 144880 } TagArray{ size = 154398 bytes = 482512 } seq expand size = 1080786 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0348 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.3792 value = 3.1069 dict = 1.00 all = 3.4997 } Zip/UnZip{ index = 0.0963 value = 0.3219 dict = 1.00 all = 0.2857 } ---------------------------- total value len = 10.557354 GB avg = 0.159 KB (by entry num) total key len = 1.830964 GB avg = 0.028 KB total ukey len = 1.276552 GB avg = 0.016 KB total ukey num = 0.079484114 Billion total entry num = 0.066291843 Billion write speed all = 0.336056580 MB/sec (with seq num) write speed all = 0.321670246 MB/sec (without seq num) 2024/11/23-19:09:36.049945 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276500: 926692 keys, 38350821 bytes, temperature: kWarm 2024/11/23-19:09:36.050031 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360176049995, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276500, "file_size": 38350821, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3273700812, "table_properties": {"data_size": 34847984, "index_size": 1785664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947376, "raw_average_key_size": 28, "raw_value_size": 108270380, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926692, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360172, "oldest_key_time": 0, "file_creation_time": 1732360172, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276500, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:36.086464 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276501.sst) /dev/shm/ToplingTemp/Topling-161098-Ce6H8F 2024/11/23-19:09:37.105785 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276501.sst): first pass time = 1.02's, 131.679'MB/sec 2024/11/23-19:09:38.438044 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276501.sst): old prealloc_size = 147639500, real_size = 35224688 2024/11/23-19:09:38.438611 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276501.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928671 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 116.53 avg-zval = 35.79 2024/11/23-19:09:38.440532 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276501.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-19:09:39.274188 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276501.sst): second pass time = 1.33's, 81.271'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.848'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3232.290'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.329'MB/sec write SST data time = 0.13's, 301.671'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32244.125'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.271'MB/sec zip pipeline throughput = 81.271'MB/sec entries = 928671 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.53 avg-zval = 35.79 usrkeys = 928671 avg-key = 20.00 avg-zkey = 1.89 avg-val = 122.87 avg-zval = 35.79 TagRS{ kinds = RS_Key0_TagN bytes = 145200 } TagArray{ size = 840931 bytes = 2943280 } seq expand size = 5886517 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.5827 value = 3.2558 dict = 1.00 all = 3.4265 } Zip/UnZip{ index = 0.0945 value = 0.3071 dict = 1.00 all = 0.2918 } ---------------------------- total value len = 10.671456 GB avg = 0.159 KB (by entry num) total key len = 1.856967 GB avg = 0.028 KB total ukey len = 1.295125 GB avg = 0.016 KB total ukey num = 0.080412785 Billion total entry num = 0.067220514 Billion write speed all = 0.339827011 MB/sec (with seq num) write speed all = 0.325240436 MB/sec (without seq num) 2024/11/23-19:09:39.321782 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276501: 928671 keys, 39170464 bytes, temperature: kWarm 2024/11/23-19:09:39.321853 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360179321816, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276501, "file_size": 39170464, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 4205696442, "table_properties": {"data_size": 33237280, "index_size": 1755072, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26002788, "raw_average_key_size": 28, "raw_value_size": 108214998, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928671, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360176, "oldest_key_time": 0, "file_creation_time": 1732360176, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276501, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:39.364426 139957559264832 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276502.sst) /dev/shm/ToplingTemp/Topling-161098-zcnFjG 2024/11/23-19:09:39.483417 139957559264832 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276502.sst): first pass time = 0.12's, 230.112'MB/sec 2024/11/23-19:09:39.743837 139957559264832 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276502.sst): old prealloc_size = 147639500, real_size = 7817040 2024/11/23-19:09:39.744042 139957559264832 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276502.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 187480 prefix = 4 raw-key = 0.0030 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0221 GB zip-val = 0.0074 GB avg-val = 118.00 avg-zval = 39.51 2024/11/23-19:09:39.744526 139957559264832 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276502.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.027 GB 2024/11/23-19:09:39.911821 139957559264832 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00/276502.sst): second pass time = 0.26's, 85.183'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.17's, 51.278'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1981.418'MB/sec (index lex order gen) rebuild zvType time = 0.14's, 0.332'MB/sec write SST data time = 0.03's, 324.731'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.232'MB, throughput = 8073.403'MB/sec zip my value time = 0.26's, unzip length = 0.022'GB zip my value throughput = 85.183'MB/sec zip pipeline throughput = 85.183'MB/sec entries = 187480 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 39.51 usrkeys = 187480 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 39.51 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 187480 bytes = 609328 } seq expand size = 1312360 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0037 GB value = 0.0221 GB dict = 0.23 MB all = 0.0274 GB } __ZipSize{ index = 0.0004 GB value = 0.0074 GB dict = 0.23 MB all = 0.0086 GB } UnZip/Zip{ index = 10.3365 value = 2.9866 dict = 1.00 all = 3.1781 } Zip/UnZip{ index = 0.0967 value = 0.3348 dict = 1.00 all = 0.3147 } ---------------------------- total value len = 10.694891 GB avg = 0.159 KB (by entry num) total key len = 1.862216 GB avg = 0.028 KB total ukey len = 1.298875 GB avg = 0.016 KB total ukey num = 0.080600265 Billion total entry num = 0.067407994 Billion write speed all = 0.340599171 MB/sec (with seq num) write speed all = 0.325972167 MB/sec (without seq num) 2024/11/23-19:09:39.928440 139957559264832 [compaction_job.cc:1948] [default] [JOB 33865] Generated table #276502: 187480 keys, 8612849 bytes, temperature: kWarm 2024/11/23-19:09:39.928498 139957559264832 EVENT_LOG_v1 {"time_micros": 1732360179928469, "cf_name": "default", "job": 33865, "event": "table_file_creation", "file_number": 276502, "file_size": 8612849, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4106697520, "largest_seqno": 4141302234, "table_properties": {"data_size": 7407232, "index_size": 362752, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5249440, "raw_average_key_size": 28, "raw_value_size": 22122640, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 187480, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732360179, "oldest_key_time": 0, "file_creation_time": 1732360179, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 276502, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-19:09:39.944619 139957559264832 [compaction_job.cc:703] job-33865: subcompact[0], size: 0.622389 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-33865/att-00 [276486,276487,276488,276489,276490,276491,276492,276493,276494,276495,276496,276497,276498,276499,276500,276501,276502] 2024/11/23-19:09:40.044197 139957559264832 (Original Log Time 2024/11/23-19:08:38.001908) EVENT_LOG_v1 {"time_micros": 1732360118001863, "job": 33865, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [276457], "files_L3": [276404, 276405, 276406, 276407, 276408, 276409, 276410, 276411, 276412, 276413, 276414, 276415, 276416, 276417, 276418, 276478], "score": 0.989923, "input_data_size": 641444132, "oldest_snapshot_seqno": 4205813777} 2024/11/23-19:09:40.044201 139957559264832 (Original Log Time 2024/11/23-19:09:40.042822) [compaction_job.cc:2022] [default] [JOB 33865] Compacted 1@2 + 16@3 files to L3 => 622389421 bytes 2024/11/23-19:09:40.044202 139957559264832 (Original Log Time 2024/11/23-19:09:40.044134) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 10.4 rd, 10.0 wr, level 3, files in(1, 16) out(17 +0 blob) MB in(27.1, 584.6 +0.0 blob) out(593.6 +0.0 blob), read-write-amplify(44.5) write-amplify(21.9) OK, records in: 15092960, records dropped: 43130 output_compression: Snappy 2024/11/23-19:09:40.044208 139957559264832 (Original Log Time 2024/11/23-19:09:40.044177) EVENT_LOG_v1 {"time_micros": 1732360180044158, "cf": "default", "job": 33865, "event": "compaction_finished", "compaction_time_micros": 61942717, "compaction_time_cpu_micros": 24645120, "output_level": 3, "num_output_files": 17, "total_output_size": 622389421, "num_input_records": 15092960, "num_output_records": 15049830, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 477371784, "file_range_sync_nanos": 0, "file_fsync_nanos": 12519, "file_prepare_write_nanos": 48951, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/23-19:09:40.180509 139957559264832 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732360117,"executesMs":62175,"compactionJobId":33865,"attempt":0,"compactionInputRawBytes":2180981528,"compactionInputZipBytes":641444132,"compactionOutputRawBytes":0,"compactionOutputZipBytes":622389421,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-46f7530423cc9243b5fa7f6266e61d11-4ec8074b12682be5-00"}}, response = 2024/11/23-19:09:40.180592 139957559264832 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-33865/att-00: olev 3, work 62.175 s, result 6.023 ms, install 1.402 ms, input{raw 2.031 GiB zip 611.729 MiB}