2024/11/22-21:10:15.958009 140024835753536 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00: fork to child time = 0.009485 sec 2024/11/22-21:10:15.958875 140024835753536 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00 2024/11/22-21:10:15.966358 140024835753536 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00 2024/11/22-21:10:15.967664 140024835753536 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-21:10:15.968002 140024835753536 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-21:10:16.024963 140024835753536 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11136/att-00/MANIFEST-206347 2024/11/22-21:10:16.038747 140024835753536 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11136/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 227961, last_sequence is 3565213929, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 227815 2024/11/22-21:10:16.038769 140024835753536 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-21:10:16.038871 140024835753536 [version_set.cc:5721] Creating manifest 227961 2024/11/22-21:10:16.071822 140024835753536 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-21:10:16.092184 140024835753536 [compaction_job.cc:2437] [default] [JOB 11136] Compacting 1@2 + 12@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/22-21:10:16.092211 140024835753536 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 8395 Base level 2, inputs: [227955(27MB)], [227886(36MB) 227887(36MB) 227888(36MB) 227889(36MB) 227906(36MB) 227907(37MB) 227908(35MB) 227909(36MB) 227910(36MB) 227911(35MB) 227912(36MB) 227913(35MB)] 2024/11/22-21:10:16.100339 140024835753536 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:10:16.100389 140024835753536 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:10:16.100417 140024835753536 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:10:16.100425 140024835753536 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-21:10:16.100439 140024835753536 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-21:10:16.612856 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227962.sst) /dev/shm/ToplingTemp/Topling-292055-R3yH80 2024/11/22-21:10:17.692287 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227962.sst): first pass time = 1.08's, 124.347'MB/sec 2024/11/22-21:10:19.629046 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227962.sst): old prealloc_size = 147639500, real_size = 33944304 2024/11/22-21:10:19.629607 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227962.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928602 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0319 GB avg-val = 116.54 avg-zval = 34.40 2024/11/22-21:10:19.630385 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227962.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:10:20.690356 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227962.sst): second pass time = 1.94's, 55.898'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.06's, 35.741'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3265.073'MB/sec (index lex order gen) rebuild zvType time = 0.93's, 0.249'MB/sec write SST data time = 0.13's, 294.337'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32242.841'MB/sec zip my value time = 1.94's, unzip length = 0.108'GB zip my value throughput = 55.898'MB/sec zip pipeline throughput = 62.574'MB/sec entries = 928602 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.54 avg-zval = 34.40 usrkeys = 928602 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.54 avg-zval = 34.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928602 bytes = 3134048 } seq expand size = 6500214 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0319 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.5144 value = 3.3875 dict = 1.00 all = 3.5384 } Zip/UnZip{ index = 0.0951 value = 0.2952 dict = 1.00 all = 0.2826 } ---------------------------- total value len = 0.114717 GB avg = 0.124 KB (by entry num) total key len = 0.026001 GB avg = 0.028 KB total ukey len = 0.018572 GB avg = 0.020 KB total ukey num = 0.000928602 Billion total entry num = 0.000928602 Billion write speed all = 0.003178088 MB/sec (with seq num) write speed all = 0.003010309 MB/sec (without seq num) 2024/11/22-21:10:20.727761 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227962: 928602 keys, 37931764 bytes, temperature: kWarm 2024/11/22-21:10:20.727847 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281020727797, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227962, "file_size": 37931764, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038037, "largest_seqno": 3565172826, "table_properties": {"data_size": 31945632, "index_size": 1766336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26000856, "raw_average_key_size": 28, "raw_value_size": 108217008, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928602, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281016, "oldest_key_time": 0, "file_creation_time": 1732281016, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227962, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:10:20.766823 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227963.sst) /dev/shm/ToplingTemp/Topling-292055-E6Ebe0 2024/11/22-21:10:21.592909 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227963.sst): first pass time = 0.83's, 162.481'MB/sec 2024/11/22-21:10:23.004580 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227963.sst): old prealloc_size = 147639500, real_size = 34473984 2024/11/22-21:10:23.005170 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227963.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927055 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0325 GB avg-val = 116.78 avg-zval = 35.02 2024/11/22-21:10:23.005926 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227963.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:10:24.298482 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227963.sst): second pass time = 1.41's, 76.757'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.29's, 29.723'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3114.045'MB/sec (index lex order gen) rebuild zvType time = 1.05's, 0.220'MB/sec write SST data time = 0.24's, 160.310'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 25286.139'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.757'MB/sec zip pipeline throughput = 76.757'MB/sec entries = 927055 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.78 avg-zval = 35.02 usrkeys = 927055 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.78 avg-zval = 35.02 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927055 bytes = 3128832 } seq expand size = 6489385 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.4455 value = 3.3345 dict = 1.00 all = 3.4900 } Zip/UnZip{ index = 0.0957 value = 0.2999 dict = 1.00 all = 0.2865 } ---------------------------- total value len = 0.229467 GB avg = 0.124 KB (by entry num) total key len = 0.051958 GB avg = 0.028 KB total ukey len = 0.037113 GB avg = 0.020 KB total ukey num = 0.001855657 Billion total entry num = 0.001855657 Billion write speed all = 0.006355411 MB/sec (with seq num) write speed all = 0.006020161 MB/sec (without seq num) 2024/11/22-21:10:24.336866 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227963: 927055 keys, 38457350 bytes, temperature: kWarm 2024/11/22-21:10:24.336947 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281024336902, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227963, "file_size": 38457350, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038075, "largest_seqno": 3565173367, "table_properties": {"data_size": 32466992, "index_size": 1775040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25957540, "raw_average_key_size": 28, "raw_value_size": 108260226, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927055, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281020, "oldest_key_time": 0, "file_creation_time": 1732281020, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227963, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:10:24.402005 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227964.sst) /dev/shm/ToplingTemp/Topling-292055-PYtbZ3 2024/11/22-21:10:25.458328 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227964.sst): first pass time = 1.06's, 127.068'MB/sec 2024/11/22-21:10:26.874343 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227964.sst): old prealloc_size = 147639500, real_size = 34468064 2024/11/22-21:10:26.874929 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227964.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928404 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0325 GB avg-val = 116.57 avg-zval = 34.97 2024/11/22-21:10:26.875683 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227964.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:10:28.173044 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227964.sst): second pass time = 1.42's, 76.463'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.30's, 29.611'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3133.352'MB/sec (index lex order gen) rebuild zvType time = 1.06's, 0.220'MB/sec write SST data time = 0.24's, 159.758'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 36733.528'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.463'MB/sec zip pipeline throughput = 76.463'MB/sec entries = 928404 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.57 avg-zval = 34.97 usrkeys = 928404 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.57 avg-zval = 34.97 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928404 bytes = 3133376 } seq expand size = 6498828 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5130 value = 3.3330 dict = 1.00 all = 3.4903 } Zip/UnZip{ index = 0.0951 value = 0.3000 dict = 1.00 all = 0.2865 } ---------------------------- total value len = 0.344188 GB avg = 0.124 KB (by entry num) total key len = 0.077954 GB avg = 0.028 KB total ukey len = 0.055681 GB avg = 0.020 KB total ukey num = 0.002784061 Billion total entry num = 0.002784061 Billion write speed all = 0.009532372 MB/sec (with seq num) write speed all = 0.009029438 MB/sec (without seq num) 2024/11/22-21:10:28.212485 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227964: 928404 keys, 38454562 bytes, temperature: kWarm 2024/11/22-21:10:28.212566 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281028212522, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227964, "file_size": 38454562, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038106, "largest_seqno": 3565173930, "table_properties": {"data_size": 32469584, "index_size": 1766208, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25995312, "raw_average_key_size": 28, "raw_value_size": 108222476, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928404, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281024, "oldest_key_time": 0, "file_creation_time": 1732281024, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227964, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:10:28.281895 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227965.sst) /dev/shm/ToplingTemp/Topling-292055-6bFWT2 2024/11/22-21:10:29.219946 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227965.sst): first pass time = 0.94's, 143.090'MB/sec 2024/11/22-21:10:31.828267 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227965.sst): old prealloc_size = 147639500, real_size = 34494304 2024/11/22-21:10:31.828950 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227965.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928720 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0325 GB avg-val = 116.52 avg-zval = 34.99 2024/11/22-21:10:31.829744 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227965.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:10:33.283907 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227965.sst): second pass time = 2.61's, 41.521'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.46's, 26.437'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2669.188'MB/sec (index lex order gen) rebuild zvType time = 1.19's, 0.195'MB/sec write SST data time = 0.27's, 144.226'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 29135.253'MB/sec zip my value time = 2.61's, unzip length = 0.108'GB zip my value throughput = 41.521'MB/sec zip pipeline throughput = 117.256'MB/sec entries = 928720 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.52 avg-zval = 34.99 usrkeys = 928720 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.52 avg-zval = 34.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928720 bytes = 3134448 } seq expand size = 6501040 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5124 value = 3.3302 dict = 1.00 all = 3.4878 } Zip/UnZip{ index = 0.0951 value = 0.3003 dict = 1.00 all = 0.2867 } ---------------------------- total value len = 0.458903 GB avg = 0.124 KB (by entry num) total key len = 0.103958 GB avg = 0.028 KB total ukey len = 0.074256 GB avg = 0.020 KB total ukey num = 0.003712781 Billion total entry num = 0.003712781 Billion write speed all = 0.012708474 MB/sec (with seq num) write speed all = 0.012037846 MB/sec (without seq num) 2024/11/22-21:10:33.319979 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227965: 928720 keys, 38481816 bytes, temperature: kWarm 2024/11/22-21:10:33.320064 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281033320019, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227965, "file_size": 38481816, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038136, "largest_seqno": 3565173114, "table_properties": {"data_size": 32495056, "index_size": 1766912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26004160, "raw_average_key_size": 28, "raw_value_size": 108213668, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928720, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281028, "oldest_key_time": 0, "file_creation_time": 1732281028, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227965, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:10:33.391927 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227966.sst) /dev/shm/ToplingTemp/Topling-292055-tm2Dt0 2024/11/22-21:10:34.609538 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227966.sst): first pass time = 1.22's, 110.234'MB/sec 2024/11/22-21:10:38.392242 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227966.sst): old prealloc_size = 147639500, real_size = 34145648 2024/11/22-21:10:38.392967 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227966.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928759 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0321 GB avg-val = 116.51 avg-zval = 34.61 2024/11/22-21:10:38.393830 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227966.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:10:39.761753 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227966.sst): second pass time = 3.78's, 28.613'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.37's, 28.017'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2561.708'MB/sec (index lex order gen) rebuild zvType time = 1.16's, 0.201'MB/sec write SST data time = 0.21's, 181.171'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 19320.069'MB/sec zip my value time = 3.78's, unzip length = 0.108'GB zip my value throughput = 28.613'MB/sec zip pipeline throughput = 105.704'MB/sec entries = 928759 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.51 avg-zval = 34.61 usrkeys = 928759 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.51 avg-zval = 34.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928759 bytes = 3366768 } seq expand size = 6501313 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.5143 value = 3.3662 dict = 1.00 all = 3.4980 } Zip/UnZip{ index = 0.0951 value = 0.2971 dict = 1.00 all = 0.2859 } ---------------------------- total value len = 0.573617 GB avg = 0.124 KB (by entry num) total key len = 0.129963 GB avg = 0.028 KB total ukey len = 0.092831 GB avg = 0.020 KB total ukey num = 0.004641540 Billion total entry num = 0.004641540 Billion write speed all = 0.015883358 MB/sec (with seq num) write speed all = 0.015045094 MB/sec (without seq num) 2024/11/22-21:10:39.794607 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227966: 928759 keys, 38369504 bytes, temperature: kWarm 2024/11/22-21:10:39.794671 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281039794638, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227966, "file_size": 38369504, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3237868609, "largest_seqno": 3565173427, "table_properties": {"data_size": 32146592, "index_size": 1766656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26005252, "raw_average_key_size": 28, "raw_value_size": 108212534, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928759, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281033, "oldest_key_time": 0, "file_creation_time": 1732281033, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227966, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:10:39.833251 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227967.sst) /dev/shm/ToplingTemp/Topling-292055-6OOZe2 2024/11/22-21:10:41.338700 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227967.sst): first pass time = 1.51's, 89.157'MB/sec 2024/11/22-21:10:43.337180 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227967.sst): old prealloc_size = 147639500, real_size = 37171984 2024/11/22-21:10:43.337755 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227967.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925796 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0352 GB avg-val = 116.98 avg-zval = 38.00 2024/11/22-21:10:43.338116 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227967.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:10:44.212009 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227967.sst): second pass time = 2.00's, 54.211'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 44.838'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3183.682'MB/sec (index lex order gen) rebuild zvType time = 0.65's, 0.355'MB/sec write SST data time = 0.22's, 175.864'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36935.358'MB/sec zip my value time = 2.00's, unzip length = 0.108'GB zip my value throughput = 54.211'MB/sec zip pipeline throughput = 54.604'MB/sec entries = 925796 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.98 avg-zval = 38.00 usrkeys = 925796 avg-key = 20.00 avg-zkey = 1.90 avg-val = 119.42 avg-zval = 38.00 TagRS{ kinds = RS_Key0_TagN bytes = 144760 } TagArray{ size = 323091 bytes = 1050064 } seq expand size = 2261637 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0352 GB dict = 1.08 MB all = 0.0392 GB } UnZip/Zip{ index = 10.5128 value = 3.0784 dict = 1.00 all = 3.4218 } Zip/UnZip{ index = 0.0951 value = 0.3248 dict = 1.00 all = 0.2922 } ---------------------------- total value len = 0.684174 GB avg = 0.123 KB (by entry num) total key len = 0.155885 GB avg = 0.028 KB total ukey len = 0.111347 GB avg = 0.020 KB total ukey num = 0.005567336 Billion total entry num = 0.005567336 Billion write speed all = 0.018962484 MB/sec (with seq num) write speed all = 0.017957121 MB/sec (without seq num) 2024/11/22-21:10:44.246214 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227967: 925796 keys, 39224827 bytes, temperature: kWarm 2024/11/22-21:10:44.246278 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281044246245, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227967, "file_size": 39224827, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3273700833, "table_properties": {"data_size": 35179072, "index_size": 1761280, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25922288, "raw_average_key_size": 28, "raw_value_size": 108295472, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 925796, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281039, "oldest_key_time": 0, "file_creation_time": 1732281039, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227967, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:10:44.281099 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227968.sst) /dev/shm/ToplingTemp/Topling-292055-bZ3Ni0 2024/11/22-21:10:45.021269 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227968.sst): first pass time = 0.74's, 181.344'MB/sec 2024/11/22-21:10:46.762228 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227968.sst): old prealloc_size = 147639500, real_size = 36475232 2024/11/22-21:10:46.762821 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227968.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927090 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.77 avg-zval = 37.20 2024/11/22-21:10:46.762868 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227968.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:10:47.436307 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227968.sst): second pass time = 1.74's, 62.223'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.67's, 55.387'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3065.395'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.447'MB/sec write SST data time = 0.16's, 240.137'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 50347.206'MB/sec zip my value time = 1.74's, unzip length = 0.108'GB zip my value throughput = 62.223'MB/sec zip pipeline throughput = 62.373'MB/sec entries = 927090 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 37.20 usrkeys = 927090 avg-key = 20.00 avg-zkey = 1.89 avg-val = 116.77 avg-zval = 37.20 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 7 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5821 value = 3.1388 dict = 1.00 all = 3.5951 } Zip/UnZip{ index = 0.0945 value = 0.3186 dict = 1.00 all = 0.2782 } ---------------------------- total value len = 0.792433 GB avg = 0.122 KB (by entry num) total key len = 0.181844 GB avg = 0.028 KB total ukey len = 0.129889 GB avg = 0.020 KB total ukey num = 0.006494426 Billion total entry num = 0.006494426 Billion write speed all = 0.021990555 MB/sec (with seq num) write speed all = 0.020817861 MB/sec (without seq num) 2024/11/22-21:10:47.475491 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227968: 927090 keys, 37333230 bytes, temperature: kWarm 2024/11/22-21:10:47.475575 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281047475529, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227968, "file_size": 37333230, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34491088, "index_size": 1752192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25958520, "raw_average_key_size": 28, "raw_value_size": 108259288, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927090, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281044, "oldest_key_time": 0, "file_creation_time": 1732281044, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227968, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:10:47.545293 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227969.sst) /dev/shm/ToplingTemp/Topling-292055-P40Lz1 2024/11/22-21:10:48.562511 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227969.sst): first pass time = 1.02's, 131.955'MB/sec 2024/11/22-21:10:50.368218 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227969.sst): old prealloc_size = 147639500, real_size = 36598176 2024/11/22-21:10:50.368803 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227969.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924629 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 117.16 avg-zval = 37.43 2024/11/22-21:10:50.369108 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227969.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:10:51.347040 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227969.sst): second pass time = 1.81's, 60.015'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.98's, 39.335'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3110.934'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.330'MB/sec write SST data time = 0.28's, 138.833'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35748.826'MB/sec zip my value time = 1.81's, unzip length = 0.108'GB zip my value throughput = 60.015'MB/sec zip pipeline throughput = 60.015'MB/sec entries = 924629 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.16 avg-zval = 37.43 usrkeys = 924629 avg-key = 20.00 avg-zkey = 1.90 avg-val = 119.25 avg-zval = 37.43 TagRS{ kinds = RS_Key0_TagN bytes = 144560 } TagArray{ size = 276653 bytes = 899136 } seq expand size = 1936571 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5121 value = 3.1302 dict = 1.00 all = 3.4861 } Zip/UnZip{ index = 0.0951 value = 0.3195 dict = 1.00 all = 0.2869 } ---------------------------- total value len = 0.902698 GB avg = 0.122 KB (by entry num) total key len = 0.207734 GB avg = 0.028 KB total ukey len = 0.148381 GB avg = 0.020 KB total ukey num = 0.007419055 Billion total entry num = 0.007419055 Billion write speed all = 0.025061504 MB/sec (with seq num) write speed all = 0.023721969 MB/sec (without seq num) 2024/11/22-21:10:51.382410 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227969: 924629 keys, 38501355 bytes, temperature: kWarm 2024/11/22-21:10:51.382472 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281051382441, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227969, "file_size": 38501355, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3330445142, "table_properties": {"data_size": 34607696, "index_size": 1759168, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25889612, "raw_average_key_size": 28, "raw_value_size": 108328190, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924629, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281047, "oldest_key_time": 0, "file_creation_time": 1732281047, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227969, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:10:51.420734 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227970.sst) /dev/shm/ToplingTemp/Topling-292055-lMjAa1 2024/11/22-21:10:52.391538 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227970.sst): first pass time = 0.97's, 138.260'MB/sec 2024/11/22-21:10:53.961475 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227970.sst): old prealloc_size = 147639500, real_size = 34861280 2024/11/22-21:10:53.962018 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227970.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924558 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0329 GB avg-val = 117.17 avg-zval = 35.55 2024/11/22-21:10:53.962806 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227970.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:10:55.037373 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227970.sst): second pass time = 1.57's, 69.057'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.08's, 36.206'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3391.878'MB/sec (index lex order gen) rebuild zvType time = 0.95's, 0.244'MB/sec write SST data time = 0.13's, 302.759'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33687.798'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 69.057'MB/sec zip pipeline throughput = 69.057'MB/sec entries = 924558 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.17 avg-zval = 35.55 usrkeys = 924558 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.17 avg-zval = 35.55 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924558 bytes = 3235968 } seq expand size = 6471906 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.5105 value = 3.2956 dict = 1.00 all = 3.4456 } Zip/UnZip{ index = 0.0951 value = 0.3034 dict = 1.00 all = 0.2902 } ---------------------------- total value len = 1.017500 GB avg = 0.122 KB (by entry num) total key len = 0.233621 GB avg = 0.028 KB total ukey len = 0.166872 GB avg = 0.020 KB total ukey num = 0.008343613 Billion total entry num = 0.008343613 Billion write speed all = 0.028234399 MB/sec (with seq num) write speed all = 0.026728058 MB/sec (without seq num) 2024/11/22-21:10:55.071451 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227970: 924558 keys, 38953364 bytes, temperature: kWarm 2024/11/22-21:10:55.071515 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281055071481, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227970, "file_size": 38953364, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3330445143, "largest_seqno": 3565173875, "table_properties": {"data_size": 32870672, "index_size": 1759296, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25887624, "raw_average_key_size": 28, "raw_value_size": 108330132, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924558, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281051, "oldest_key_time": 0, "file_creation_time": 1732281051, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227970, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:10:55.132272 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227971.sst) /dev/shm/ToplingTemp/Topling-292055-Y8OoW3 2024/11/22-21:10:56.132350 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227971.sst): first pass time = 1.00's, 134.214'MB/sec 2024/11/22-21:10:57.491230 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227971.sst): old prealloc_size = 147639500, real_size = 33681008 2024/11/22-21:10:57.491766 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227971.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924029 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0317 GB avg-val = 117.25 avg-zval = 34.28 2024/11/22-21:10:57.492602 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227971.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:10:58.433725 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227971.sst): second pass time = 1.36's, 79.799'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 40.200'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3364.585'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.284'MB/sec write SST data time = 0.13's, 294.530'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 34504.336'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.799'MB/sec zip pipeline throughput = 79.799'MB/sec entries = 924029 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.25 avg-zval = 34.28 usrkeys = 924029 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.25 avg-zval = 34.28 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924029 bytes = 3349616 } seq expand size = 6468203 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.4434 value = 3.4200 dict = 1.00 all = 3.5425 } Zip/UnZip{ index = 0.0958 value = 0.2924 dict = 1.00 all = 0.2823 } ---------------------------- total value len = 1.132313 GB avg = 0.122 KB (by entry num) total key len = 0.259494 GB avg = 0.028 KB total ukey len = 0.185353 GB avg = 0.020 KB total ukey num = 0.009267642 Billion total entry num = 0.009267642 Billion write speed all = 0.031406893 MB/sec (with seq num) write speed all = 0.029733857 MB/sec (without seq num) 2024/11/22-21:10:58.469707 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227971: 924029 keys, 37887650 bytes, temperature: kWarm 2024/11/22-21:10:58.469770 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281058469739, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227971, "file_size": 37887650, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701386, "largest_seqno": 3565173464, "table_properties": {"data_size": 31680224, "index_size": 1769600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25872812, "raw_average_key_size": 28, "raw_value_size": 108345002, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924029, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281055, "oldest_key_time": 0, "file_creation_time": 1732281055, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227971, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:10:58.503716 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227972.sst) /dev/shm/ToplingTemp/Topling-292055-ayXZ33 2024/11/22-21:10:59.414829 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227972.sst): first pass time = 0.91's, 147.318'MB/sec 2024/11/22-21:11:00.787936 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227972.sst): old prealloc_size = 147639500, real_size = 33218816 2024/11/22-21:11:00.788511 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227972.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924260 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0312 GB avg-val = 117.22 avg-zval = 33.79 2024/11/22-21:11:00.789296 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227972.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:11:01.703562 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227972.sst): second pass time = 1.37's, 78.937'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 40.749'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3175.341'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 292.486'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 35016.574'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 78.937'MB/sec zip pipeline throughput = 78.937'MB/sec entries = 924260 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.22 avg-zval = 33.79 usrkeys = 924260 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.22 avg-zval = 33.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924260 bytes = 3234928 } seq expand size = 6469820 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5106 value = 3.4692 dict = 1.00 all = 3.5973 } Zip/UnZip{ index = 0.0951 value = 0.2883 dict = 1.00 all = 0.2780 } ---------------------------- total value len = 1.247121 GB avg = 0.122 KB (by entry num) total key len = 0.285373 GB avg = 0.028 KB total ukey len = 0.203838 GB avg = 0.020 KB total ukey num = 0.010191902 Billion total entry num = 0.010191902 Billion write speed all = 0.034579035 MB/sec (with seq num) write speed all = 0.032739284 MB/sec (without seq num) 2024/11/22-21:11:01.736932 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227972: 924260 keys, 37310228 bytes, temperature: kWarm 2024/11/22-21:11:01.736998 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281061736965, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227972, "file_size": 37310228, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3378289821, "largest_seqno": 3565173479, "table_properties": {"data_size": 31228848, "index_size": 1758720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25879280, "raw_average_key_size": 28, "raw_value_size": 108338520, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924260, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281058, "oldest_key_time": 0, "file_creation_time": 1732281058, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227972, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:11:01.772946 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227973.sst) /dev/shm/ToplingTemp/Topling-292055-frxPl3 2024/11/22-21:11:02.714293 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227973.sst): first pass time = 0.94's, 142.587'MB/sec 2024/11/22-21:11:04.067833 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227973.sst): old prealloc_size = 147639500, real_size = 32039904 2024/11/22-21:11:04.068381 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227973.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924028 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0301 GB avg-val = 117.25 avg-zval = 32.52 2024/11/22-21:11:04.069143 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227973.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:11:04.935424 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227973.sst): second pass time = 1.35's, 80.081'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 41.646'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3355.306'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.313'MB/sec write SST data time = 0.13's, 281.279'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 33888.346'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.081'MB/sec zip pipeline throughput = 80.081'MB/sec entries = 924028 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.25 avg-zval = 32.52 usrkeys = 924028 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.25 avg-zval = 32.52 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924028 bytes = 3234112 } seq expand size = 6468196 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0301 GB dict = 1.08 MB all = 0.0361 GB } UnZip/Zip{ index = 10.5122 value = 3.6054 dict = 1.00 all = 3.7147 } Zip/UnZip{ index = 0.0951 value = 0.2774 dict = 1.00 all = 0.2692 } ---------------------------- total value len = 1.361935 GB avg = 0.123 KB (by entry num) total key len = 0.311246 GB avg = 0.028 KB total ukey len = 0.222319 GB avg = 0.020 KB total ukey num = 0.011115930 Billion total entry num = 0.011115930 Billion write speed all = 0.037750705 MB/sec (with seq num) write speed all = 0.035744302 MB/sec (without seq num) 2024/11/22-21:11:04.966802 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227973: 924028 keys, 36131174 bytes, temperature: kWarm 2024/11/22-21:11:04.966865 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281064966833, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227973, "file_size": 36131174, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774735, "largest_seqno": 3565173495, "table_properties": {"data_size": 30050704, "index_size": 1758016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25872784, "raw_average_key_size": 28, "raw_value_size": 108345004, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924028, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281061, "oldest_key_time": 0, "file_creation_time": 1732281061, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227973, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:11:05.004714 140024835753536 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227974.sst) /dev/shm/ToplingTemp/Topling-292055-v1Y073 2024/11/22-21:11:05.089971 140024835753536 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227974.sst): first pass time = 0.09's, 235.501'MB/sec 2024/11/22-21:11:05.273372 140024835753536 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227974.sst): old prealloc_size = 147639500, real_size = 5906992 2024/11/22-21:11:05.273502 140024835753536 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227974.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 137454 prefix = 4 raw-key = 0.0022 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0162 GB zip-val = 0.0056 GB avg-val = 118.00 avg-zval = 40.80 2024/11/22-21:11:05.273633 140024835753536 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227974.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.020 GB 2024/11/22-21:11:05.402927 140024835753536 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00/227974.sst): second pass time = 0.18's, 88.728'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.13's, 49.974'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2499.456'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.313'MB/sec write SST data time = 0.02's, 330.109'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.170'MB, throughput = 5809.032'MB/sec zip my value time = 0.18's, unzip length = 0.016'GB zip my value throughput = 88.728'MB/sec zip pipeline throughput = 88.728'MB/sec entries = 137454 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 40.80 usrkeys = 137454 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 40.80 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 137454 bytes = 429552 } seq expand size = 962178 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0027 GB value = 0.0162 GB dict = 0.17 MB all = 0.0201 GB } __ZipSize{ index = 0.0003 GB value = 0.0056 GB dict = 0.17 MB all = 0.0065 GB } UnZip/Zip{ index = 10.4006 value = 2.8922 dict = 1.00 all = 3.1000 } Zip/UnZip{ index = 0.0961 value = 0.3458 dict = 1.00 all = 0.3226 } ---------------------------- total value len = 1.379116 GB avg = 0.123 KB (by entry num) total key len = 0.315095 GB avg = 0.028 KB total ukey len = 0.225068 GB avg = 0.020 KB total ukey num = 0.011253384 Billion total entry num = 0.011253384 Billion write speed all = 0.038224796 MB/sec (with seq num) write speed all = 0.036193605 MB/sec (without seq num) 2024/11/22-21:11:05.415603 140024835753536 [compaction_job.cc:1948] [default] [JOB 11136] Generated table #227974: 137454 keys, 6473695 bytes, temperature: kWarm 2024/11/22-21:11:05.415661 140024835753536 EVENT_LOG_v1 {"time_micros": 1732281065415631, "cf_name": "default", "job": 11136, "event": "table_file_creation", "file_number": 227974, "file_size": 6473695, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3396021850, "largest_seqno": 3423914115, "table_properties": {"data_size": 5608096, "index_size": 264320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 3848712, "raw_average_key_size": 28, "raw_value_size": 16219572, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 137454, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732281064, "oldest_key_time": 0, "file_creation_time": 1732281064, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 227974, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:11:05.429501 140024835753536 [compaction_job.cc:703] job-11136: subcompact[0], size: 0.463511 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11136/att-00 [227962,227963,227964,227965,227966,227967,227968,227969,227970,227971,227972,227973,227974] 2024/11/22-21:11:05.625392 140024835753536 (Original Log Time 2024/11/22-21:10:16.092288) EVENT_LOG_v1 {"time_micros": 1732281016092230, "job": 11136, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [227955], "files_L3": [227886, 227887, 227888, 227889, 227906, 227907, 227908, 227909, 227910, 227911, 227912, 227913], "score": 0.999345, "input_data_size": 486562954, "oldest_snapshot_seqno": 3565213860} 2024/11/22-21:11:05.625398 140024835753536 (Original Log Time 2024/11/22-21:11:05.624217) [compaction_job.cc:2022] [default] [JOB 11136] Compacted 1@2 + 12@3 files to L3 => 463510519 bytes 2024/11/22-21:11:05.625401 140024835753536 (Original Log Time 2024/11/22-21:11:05.625304) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 9.9 rd, 9.4 wr, level 3, files in(1, 12) out(13 +0 blob) MB in(27.1, 436.9 +0.0 blob) out(442.0 +0.0 blob), read-write-amplify(33.4) write-amplify(16.3) OK, records in: 11347985, records dropped: 94601 output_compression: Snappy 2024/11/22-21:11:05.625410 140024835753536 (Original Log Time 2024/11/22-21:11:05.625367) EVENT_LOG_v1 {"time_micros": 1732281065625336, "cf": "default", "job": 11136, "event": "compaction_finished", "compaction_time_micros": 49337225, "compaction_time_cpu_micros": 21015772, "output_level": 3, "num_output_files": 13, "total_output_size": 463510519, "num_input_records": 11347985, "num_output_records": 11253384, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 339679512, "file_range_sync_nanos": 0, "file_fsync_nanos": 13290, "file_prepare_write_nanos": 33395, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/22-21:11:05.771879 140024835753536 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732281015,"executesMs":49659,"compactionJobId":11136,"attempt":0,"compactionInputRawBytes":1644115186,"compactionInputZipBytes":486562954,"compactionOutputRawBytes":0,"compactionOutputZipBytes":463510519,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-8e29ee3057d39d3625126eec32883bdb-81dad64085c49736-00"}}, response = 2024/11/22-21:11:05.771955 140024835753536 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11136/att-00: olev 3, work 49.659 s, result 6.672 ms, install 1.222 ms, input{raw 1.531 GiB zip 464.023 MiB}