2024/11/23-03:16:21.870432 140024709862976 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00: fork to child time = 0.019708 sec 2024/11/23-03:16:21.871602 140024709862976 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00 2024/11/23-03:16:21.880138 140024709862976 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00 2024/11/23-03:16:21.881575 140024709862976 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/23-03:16:21.881974 140024709862976 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/23-03:16:21.930083 140024709862976 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18149/att-00/MANIFEST-206347 2024/11/23-03:16:21.945377 140024709862976 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18149/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 242322, last_sequence is 3754811562, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 242173 2024/11/23-03:16:21.945391 140024709862976 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/23-03:16:21.945523 140024709862976 [version_set.cc:5721] Creating manifest 242322 2024/11/23-03:16:21.988012 140024709862976 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/23-03:16:22.016500 140024709862976 [compaction_job.cc:2437] [default] [JOB 18149] Compacting 1@2 + 11@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/23-03:16:22.016526 140024709862976 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 13558 Base level 2, inputs: [242315(27MB)], [242257(36MB) 242258(35MB) 242259(35MB) 242260(35MB) 242261(35MB) 242263(36MB) 242264(36MB) 242265(35MB) 242266(33MB) 242267(35MB) 242270(36MB)] 2024/11/23-03:16:22.024264 140024709862976 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:16:22.024305 140024709862976 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:16:22.024333 140024709862976 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/23-03:16:22.024341 140024709862976 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-03:16:22.024356 140024709862976 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/23-03:16:22.631027 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242323.sst) /dev/shm/ToplingTemp/Topling-392149-WJJ9FF 2024/11/23-03:16:23.385491 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242323.sst): first pass time = 0.75's, 177.911'MB/sec 2024/11/23-03:16:25.042736 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242323.sst): old prealloc_size = 147639500, real_size = 34366464 2024/11/23-03:16:25.043300 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242323.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929859 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0324 GB avg-val = 116.34 avg-zval = 34.81 2024/11/23-03:16:25.044118 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242323.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:16:26.291485 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242323.sst): second pass time = 1.66's, 65.307'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.25's, 30.903'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3236.300'MB/sec (index lex order gen) rebuild zvType time = 1.11's, 0.209'MB/sec write SST data time = 0.14's, 278.868'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33392.195'MB/sec zip my value time = 1.66's, unzip length = 0.108'GB zip my value throughput = 65.307'MB/sec zip pipeline throughput = 65.307'MB/sec entries = 929859 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.34 avg-zval = 34.81 usrkeys = 929859 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.34 avg-zval = 34.81 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929859 bytes = 3370752 } seq expand size = 6509013 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.5123 value = 3.3426 dict = 1.00 all = 3.4781 } Zip/UnZip{ index = 0.0951 value = 0.2992 dict = 1.00 all = 0.2875 } ---------------------------- total value len = 0.114691 GB avg = 0.123 KB (by entry num) total key len = 0.026036 GB avg = 0.028 KB total ukey len = 0.018597 GB avg = 0.020 KB total ukey num = 0.000929859 Billion total entry num = 0.000929859 Billion write speed all = 0.002124397 MB/sec (with seq num) write speed all = 0.002012101 MB/sec (without seq num) 2024/11/23-03:16:26.326349 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242323: 929859 keys, 38589610 bytes, temperature: kWarm 2024/11/23-03:16:26.326437 140024709862976 EVENT_LOG_v1 {"time_micros": 1732302986326384, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242323, "file_size": 38589610, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038037, "largest_seqno": 3754791102, "table_properties": {"data_size": 32364720, "index_size": 1769088, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26036052, "raw_average_key_size": 28, "raw_value_size": 108181782, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929859, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732302982, "oldest_key_time": 0, "file_creation_time": 1732302982, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242323, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:16:26.375038 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242324.sst) /dev/shm/ToplingTemp/Topling-392149-cMQ9CH 2024/11/23-03:16:27.529982 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242324.sst): first pass time = 1.15's, 116.216'MB/sec 2024/11/23-03:16:29.092052 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242324.sst): old prealloc_size = 147639500, real_size = 33650576 2024/11/23-03:16:29.092736 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242324.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926920 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0316 GB avg-val = 116.80 avg-zval = 34.14 2024/11/23-03:16:29.093547 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242324.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:16:30.017668 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242324.sst): second pass time = 1.56's, 69.344'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.909'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2680.577'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.296'MB/sec write SST data time = 0.14's, 265.765'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32844.973'MB/sec zip my value time = 1.56's, unzip length = 0.108'GB zip my value throughput = 69.344'MB/sec zip pipeline throughput = 69.344'MB/sec entries = 926920 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.80 avg-zval = 34.14 usrkeys = 926920 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.80 avg-zval = 34.14 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926920 bytes = 3360096 } seq expand size = 6488440 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0316 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.4447 value = 3.4213 dict = 1.00 all = 3.5446 } Zip/UnZip{ index = 0.0957 value = 0.2923 dict = 1.00 all = 0.2821 } ---------------------------- total value len = 0.229443 GB avg = 0.124 KB (by entry num) total key len = 0.051990 GB avg = 0.028 KB total ukey len = 0.037136 GB avg = 0.020 KB total ukey num = 0.001856779 Billion total entry num = 0.001856779 Billion write speed all = 0.004248245 MB/sec (with seq num) write speed all = 0.004024020 MB/sec (without seq num) 2024/11/23-03:16:30.051530 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242324: 926920 keys, 37865088 bytes, temperature: kWarm 2024/11/23-03:16:30.051595 140024709862976 EVENT_LOG_v1 {"time_micros": 1732302990051557, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242324, "file_size": 37865088, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038075, "largest_seqno": 3754791790, "table_properties": {"data_size": 31643776, "index_size": 1774912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25953760, "raw_average_key_size": 28, "raw_value_size": 108264104, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926920, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732302986, "oldest_key_time": 0, "file_creation_time": 1732302986, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242324, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:16:30.088369 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242325.sst) /dev/shm/ToplingTemp/Topling-392149-ZA4TsG 2024/11/23-03:16:31.260630 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242325.sst): first pass time = 1.17's, 114.499'MB/sec 2024/11/23-03:16:32.993483 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242325.sst): old prealloc_size = 147639500, real_size = 33329568 2024/11/23-03:16:32.994078 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242325.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929910 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0313 GB avg-val = 116.33 avg-zval = 33.66 2024/11/23-03:16:32.994880 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242325.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:16:34.068053 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242325.sst): second pass time = 1.73's, 62.455'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.07's, 34.948'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3109.052'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.277'MB/sec write SST data time = 0.23's, 160.525'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31828.342'MB/sec zip my value time = 1.73's, unzip length = 0.108'GB zip my value throughput = 62.455'MB/sec zip pipeline throughput = 94.295'MB/sec entries = 929910 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.33 avg-zval = 33.66 usrkeys = 929910 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.33 avg-zval = 33.66 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929910 bytes = 3370944 } seq expand size = 6509370 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.3781 value = 3.4557 dict = 1.00 all = 3.5741 } Zip/UnZip{ index = 0.0964 value = 0.2894 dict = 1.00 all = 0.2798 } ---------------------------- total value len = 0.344133 GB avg = 0.123 KB (by entry num) total key len = 0.078027 GB avg = 0.028 KB total ukey len = 0.055734 GB avg = 0.020 KB total ukey num = 0.002786689 Billion total entry num = 0.002786689 Billion write speed all = 0.006372137 MB/sec (with seq num) write speed all = 0.006035636 MB/sec (without seq num) 2024/11/23-03:16:34.106280 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242325: 929910 keys, 37552630 bytes, temperature: kWarm 2024/11/23-03:16:34.106602 140024709862976 EVENT_LOG_v1 {"time_micros": 1732302994106317, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242325, "file_size": 37552630, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038106, "largest_seqno": 3754791822, "table_properties": {"data_size": 31304848, "index_size": 1792064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26037480, "raw_average_key_size": 28, "raw_value_size": 108180252, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929910, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732302990, "oldest_key_time": 0, "file_creation_time": 1732302990, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242325, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:16:34.163186 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242326.sst) /dev/shm/ToplingTemp/Topling-392149-cB2zRG 2024/11/23-03:16:35.236817 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242326.sst): first pass time = 1.07's, 125.018'MB/sec 2024/11/23-03:16:36.573541 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242326.sst): old prealloc_size = 147639500, real_size = 32900816 2024/11/23-03:16:36.574078 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242326.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926689 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0309 GB avg-val = 116.84 avg-zval = 33.35 2024/11/23-03:16:36.574834 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242326.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:16:37.486889 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242326.sst): second pass time = 1.34's, 81.034'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 40.510'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3389.241'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.295'MB/sec write SST data time = 0.13's, 290.421'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33962.597'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 81.034'MB/sec zip pipeline throughput = 81.034'MB/sec entries = 926689 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 33.35 usrkeys = 926689 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.84 avg-zval = 33.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926689 bytes = 3243424 } seq expand size = 6486823 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0309 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5130 value = 3.5032 dict = 1.00 all = 3.6276 } Zip/UnZip{ index = 0.0951 value = 0.2855 dict = 1.00 all = 0.2757 } ---------------------------- total value len = 0.458890 GB avg = 0.124 KB (by entry num) total key len = 0.103975 GB avg = 0.028 KB total ukey len = 0.074268 GB avg = 0.020 KB total ukey num = 0.003713378 Billion total entry num = 0.003713378 Billion write speed all = 0.008495510 MB/sec (with seq num) write speed all = 0.008047131 MB/sec (without seq num) 2024/11/23-03:16:37.522858 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242326: 926689 keys, 36998906 bytes, temperature: kWarm 2024/11/23-03:16:37.522919 140024709862976 EVENT_LOG_v1 {"time_micros": 1732302997522889, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242326, "file_size": 36998906, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3522904895, "largest_seqno": 3754791443, "table_properties": {"data_size": 30905984, "index_size": 1762944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947292, "raw_average_key_size": 28, "raw_value_size": 108270450, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926689, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732302994, "oldest_key_time": 0, "file_creation_time": 1732302994, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242326, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:16:37.576049 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242327.sst) /dev/shm/ToplingTemp/Topling-392149-5H2QHE 2024/11/23-03:16:38.871977 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242327.sst): first pass time = 1.30's, 103.572'MB/sec 2024/11/23-03:16:40.889055 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242327.sst): old prealloc_size = 147639500, real_size = 33199952 2024/11/23-03:16:40.889741 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242327.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928353 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0312 GB avg-val = 116.58 avg-zval = 33.61 2024/11/23-03:16:40.890566 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242327.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:16:42.077394 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242327.sst): second pass time = 2.00's, 54.058'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.19's, 31.489'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2649.576'MB/sec (index lex order gen) rebuild zvType time = 1.02's, 0.227'MB/sec write SST data time = 0.17's, 226.776'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31106.623'MB/sec zip my value time = 2.00's, unzip length = 0.108'GB zip my value throughput = 54.058'MB/sec zip pipeline throughput = 66.020'MB/sec entries = 928353 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.58 avg-zval = 33.61 usrkeys = 928353 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.58 avg-zval = 33.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928353 bytes = 3365296 } seq expand size = 6498471 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5128 value = 3.4685 dict = 1.00 all = 3.5869 } Zip/UnZip{ index = 0.0951 value = 0.2883 dict = 1.00 all = 0.2788 } ---------------------------- total value len = 0.573613 GB avg = 0.124 KB (by entry num) total key len = 0.129968 GB avg = 0.028 KB total ukey len = 0.092835 GB avg = 0.020 KB total ukey num = 0.004641731 Billion total entry num = 0.004641731 Billion write speed all = 0.010618652 MB/sec (with seq num) write speed all = 0.010058217 MB/sec (without seq num) 2024/11/23-03:16:42.111069 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242327: 928353 keys, 37418702 bytes, temperature: kWarm 2024/11/23-03:16:42.111133 140024709862976 EVENT_LOG_v1 {"time_micros": 1732303002111101, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242327, "file_size": 37418702, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3434038136, "largest_seqno": 3754791532, "table_properties": {"data_size": 31201536, "index_size": 1766144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25993884, "raw_average_key_size": 28, "raw_value_size": 108223914, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928353, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732302997, "oldest_key_time": 0, "file_creation_time": 1732302997, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242327, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:16:42.145949 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242328.sst) /dev/shm/ToplingTemp/Topling-392149-18YEzE 2024/11/23-03:16:43.436030 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242328.sst): first pass time = 1.29's, 104.042'MB/sec 2024/11/23-03:16:45.680590 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242328.sst): old prealloc_size = 147639500, real_size = 36681920 2024/11/23-03:16:45.681296 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242328.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931032 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0347 GB avg-val = 116.16 avg-zval = 37.23 2024/11/23-03:16:45.681687 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242328.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:16:46.464788 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242328.sst): second pass time = 2.24's, 48.199'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 49.332'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2608.491'MB/sec (index lex order gen) rebuild zvType time = 0.63's, 0.371'MB/sec write SST data time = 0.16's, 248.886'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 34350.347'MB/sec zip my value time = 2.24's, unzip length = 0.108'GB zip my value throughput = 48.199'MB/sec zip pipeline throughput = 48.199'MB/sec entries = 931032 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.16 avg-zval = 37.23 usrkeys = 931032 avg-key = 20.00 avg-zkey = 1.91 avg-val = 118.24 avg-zval = 37.23 TagRS{ kinds = RS_Key0_TagN bytes = 145560 } TagArray{ size = 276649 bytes = 1002864 } seq expand size = 1936543 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0347 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.4454 value = 3.1197 dict = 1.00 all = 3.4695 } Zip/UnZip{ index = 0.0957 value = 0.3205 dict = 1.00 all = 0.2882 } ---------------------------- total value len = 0.683698 GB avg = 0.123 KB (by entry num) total key len = 0.156037 GB avg = 0.028 KB total ukey len = 0.111455 GB avg = 0.020 KB total ukey num = 0.005572763 Billion total entry num = 0.005572763 Billion write speed all = 0.012672694 MB/sec (with seq num) write speed all = 0.011999893 MB/sec (without seq num) 2024/11/23-03:16:46.498447 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242328: 931032 keys, 38685303 bytes, temperature: kWarm 2024/11/23-03:16:46.498514 140024709862976 EVENT_LOG_v1 {"time_micros": 1732303006498479, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242328, "file_size": 38685303, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3754791505, "table_properties": {"data_size": 34666352, "index_size": 1782656, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26068896, "raw_average_key_size": 28, "raw_value_size": 108148900, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931032, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303002, "oldest_key_time": 0, "file_creation_time": 1732303002, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242328, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:16:46.536814 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242329.sst) /dev/shm/ToplingTemp/Topling-392149-LIvI0D 2024/11/23-03:16:47.580744 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242329.sst): first pass time = 1.04's, 128.576'MB/sec 2024/11/23-03:16:49.641596 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242329.sst): old prealloc_size = 147639500, real_size = 36420976 2024/11/23-03:16:49.642178 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242329.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927006 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.79 avg-zval = 37.14 2024/11/23-03:16:49.642225 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242329.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:16:50.287260 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242329.sst): second pass time = 2.06's, 52.550'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.65's, 57.738'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3160.610'MB/sec (index lex order gen) rebuild zvType time = 0.52's, 0.446'MB/sec write SST data time = 0.13's, 297.424'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34380.076'MB/sec zip my value time = 2.06's, unzip length = 0.108'GB zip my value throughput = 52.550'MB/sec zip pipeline throughput = 121.638'MB/sec entries = 927006 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.14 usrkeys = 927006 avg-key = 20.00 avg-zkey = 1.90 avg-val = 116.79 avg-zval = 37.14 TagRS{ kinds = LegacyZvType bytes = 0 } TagArray{ size = 0 bytes = 0 } seq expand size = 0 multi value expand size = 0 cnt WriteAppend = 9 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5135 value = 3.1448 dict = 1.00 all = 3.6004 } Zip/UnZip{ index = 0.0951 value = 0.3180 dict = 1.00 all = 0.2777 } ---------------------------- total value len = 0.791960 GB avg = 0.122 KB (by entry num) total key len = 0.181994 GB avg = 0.028 KB total ukey len = 0.129995 GB avg = 0.020 KB total ukey num = 0.006499769 Billion total entry num = 0.006499769 Billion write speed all = 0.014697367 MB/sec (with seq num) write speed all = 0.013912693 MB/sec (without seq num) 2024/11/23-03:16:50.321765 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242329: 927006 keys, 37278944 bytes, temperature: kWarm 2024/11/23-03:16:50.321829 140024709862976 EVENT_LOG_v1 {"time_micros": 1732303010321799, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242329, "file_size": 37278944, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 0, "table_properties": {"data_size": 34425568, "index_size": 1763456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25956168, "raw_average_key_size": 28, "raw_value_size": 108261668, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927006, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "TooZip", "compression_options": "LegacyZvType-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303006, "oldest_key_time": 0, "file_creation_time": 1732303006, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242329, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:16:50.359016 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242330.sst) /dev/shm/ToplingTemp/Topling-392149-S3S7xH 2024/11/23-03:16:51.197093 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242330.sst): first pass time = 0.84's, 160.156'MB/sec 2024/11/23-03:16:52.785420 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242330.sst): old prealloc_size = 147639500, real_size = 32398032 2024/11/23-03:16:52.786009 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242330.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927585 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.1082 GB zip-val = 0.0304 GB avg-val = 116.70 avg-zval = 32.79 2024/11/23-03:16:52.786793 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242330.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:16:53.833202 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242330.sst): second pass time = 1.59's, 68.180'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.05's, 34.921'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3078.150'MB/sec (index lex order gen) rebuild zvType time = 0.92's, 0.253'MB/sec write SST data time = 0.13's, 279.167'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35238.558'MB/sec zip my value time = 1.59's, unzip length = 0.108'GB zip my value throughput = 68.180'MB/sec zip pipeline throughput = 81.123'MB/sec entries = 927585 avg-key = 28.00 avg-zkey = 1.89 avg-val = 116.70 avg-zval = 32.79 usrkeys = 927585 avg-key = 20.00 avg-zkey = 1.89 avg-val = 123.33 avg-zval = 32.79 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 879483 bytes = 3188144 } seq expand size = 6156381 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0366 GB } UnZip/Zip{ index = 10.5815 value = 3.5592 dict = 1.00 all = 3.6682 } Zip/UnZip{ index = 0.0945 value = 0.2810 dict = 1.00 all = 0.2726 } ---------------------------- total value len = 0.906361 GB avg = 0.122 KB (by entry num) total key len = 0.207966 GB avg = 0.028 KB total ukey len = 0.148547 GB avg = 0.020 KB total ukey num = 0.007427354 Billion total entry num = 0.007427354 Billion write speed all = 0.016814773 MB/sec (with seq num) write speed all = 0.015918165 MB/sec (without seq num) 2024/11/23-03:16:53.867538 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242330: 927585 keys, 36589118 bytes, temperature: kWarm 2024/11/23-03:16:53.867604 140024709862976 EVENT_LOG_v1 {"time_micros": 1732303013867571, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242330, "file_size": 36589118, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3754791045, "table_properties": {"data_size": 30412736, "index_size": 1753216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25972380, "raw_average_key_size": 28, "raw_value_size": 108245350, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927585, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303010, "oldest_key_time": 0, "file_creation_time": 1732303010, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242330, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:16:53.903894 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242331.sst) /dev/shm/ToplingTemp/Topling-392149-PHmkvE 2024/11/23-03:16:54.730181 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242331.sst): first pass time = 0.83's, 162.442'MB/sec 2024/11/23-03:16:56.059220 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242331.sst): old prealloc_size = 147639500, real_size = 30845072 2024/11/23-03:16:56.059756 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242331.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925271 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0288 GB avg-val = 117.06 avg-zval = 31.17 2024/11/23-03:16:56.060548 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242331.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:16:57.124776 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242331.sst): second pass time = 1.33's, 81.539'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.07's, 32.789'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3518.810'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.304'MB/sec write SST data time = 0.30's, 115.350'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33505.936'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.539'MB/sec zip pipeline throughput = 81.539'MB/sec entries = 925271 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.06 avg-zval = 31.17 usrkeys = 925271 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.06 avg-zval = 31.17 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925271 bytes = 3238464 } seq expand size = 6476897 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0288 GB dict = 1.08 MB all = 0.0349 GB } UnZip/Zip{ index = 10.4438 value = 3.7553 dict = 1.00 all = 3.8417 } Zip/UnZip{ index = 0.0958 value = 0.2663 dict = 1.00 all = 0.2603 } ---------------------------- total value len = 1.021149 GB avg = 0.122 KB (by entry num) total key len = 0.233874 GB avg = 0.028 KB total ukey len = 0.167052 GB avg = 0.020 KB total ukey num = 0.008352625 Billion total entry num = 0.008352625 Billion write speed all = 0.018936861 MB/sec (with seq num) write speed all = 0.017928608 MB/sec (without seq num) 2024/11/23-03:16:57.158732 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242331: 925271 keys, 34937076 bytes, temperature: kWarm 2024/11/23-03:16:57.159732 140024709862976 EVENT_LOG_v1 {"time_micros": 1732303017158772, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242331, "file_size": 34937076, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3527089527, "largest_seqno": 3754790609, "table_properties": {"data_size": 28841664, "index_size": 1771904, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25907588, "raw_average_key_size": 28, "raw_value_size": 108310226, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925271, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303013, "oldest_key_time": 0, "file_creation_time": 1732303013, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242331, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:16:57.230787 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242332.sst) /dev/shm/ToplingTemp/Topling-392149-ha23lH 2024/11/23-03:16:58.356719 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242332.sst): first pass time = 1.13's, 119.211'MB/sec 2024/11/23-03:16:59.897579 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242332.sst): old prealloc_size = 147639500, real_size = 33456976 2024/11/23-03:16:59.898218 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242332.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925381 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 117.04 avg-zval = 33.99 2024/11/23-03:16:59.900722 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242332.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:17:00.989081 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242332.sst): second pass time = 1.54's, 70.322'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.09's, 34.298'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2857.515'MB/sec (index lex order gen) rebuild zvType time = 0.96's, 0.241'MB/sec write SST data time = 0.13's, 283.603'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 39939.172'MB/sec zip my value time = 1.54's, unzip length = 0.108'GB zip my value throughput = 70.322'MB/sec zip pipeline throughput = 78.734'MB/sec entries = 925381 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.04 avg-zval = 33.99 usrkeys = 925381 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.04 avg-zval = 33.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925381 bytes = 3123168 } seq expand size = 6477667 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.4447 value = 3.4434 dict = 1.00 all = 3.5853 } Zip/UnZip{ index = 0.0957 value = 0.2904 dict = 1.00 all = 0.2789 } ---------------------------- total value len = 1.135933 GB avg = 0.122 KB (by entry num) total key len = 0.259784 GB avg = 0.028 KB total ukey len = 0.185560 GB avg = 0.020 KB total ukey num = 0.009278006 Billion total entry num = 0.009278006 Billion write speed all = 0.021058568 MB/sec (with seq num) write speed all = 0.019938676 MB/sec (without seq num) 2024/11/23-03:17:01.035129 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242332: 925381 keys, 37435394 bytes, temperature: kWarm 2024/11/23-03:17:01.035196 140024709862976 EVENT_LOG_v1 {"time_micros": 1732303021035157, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242332, "file_size": 37435394, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3631595348, "largest_seqno": 3754790997, "table_properties": {"data_size": 31453504, "index_size": 1771968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25910668, "raw_average_key_size": 28, "raw_value_size": 108307070, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925381, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303017, "oldest_key_time": 0, "file_creation_time": 1732303017, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242332, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:17:01.075101 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242333.sst) /dev/shm/ToplingTemp/Topling-392149-5pJ2DG 2024/11/23-03:17:01.981681 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242333.sst): first pass time = 0.91's, 148.055'MB/sec 2024/11/23-03:17:03.594604 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242333.sst): old prealloc_size = 147639500, real_size = 34264112 2024/11/23-03:17:03.595158 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242333.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929434 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0323 GB avg-val = 116.41 avg-zval = 34.70 2024/11/23-03:17:03.597741 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242333.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/23-03:17:04.527694 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242333.sst): second pass time = 1.61's, 67.106'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 41.122'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3336.576'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 297.121'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31060.056'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.106'MB/sec zip pipeline throughput = 69.249'MB/sec entries = 929434 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.41 avg-zval = 34.70 usrkeys = 929434 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.41 avg-zval = 34.70 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929434 bytes = 3253040 } seq expand size = 6506038 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0323 GB dict = 1.08 MB all = 0.0384 GB } UnZip/Zip{ index = 10.4440 value = 3.3547 dict = 1.00 all = 3.4980 } Zip/UnZip{ index = 0.0957 value = 0.2981 dict = 1.00 all = 0.2859 } ---------------------------- total value len = 1.250633 GB avg = 0.123 KB (by entry num) total key len = 0.285808 GB avg = 0.028 KB total ukey len = 0.204149 GB avg = 0.020 KB total ukey num = 0.010207440 Billion total entry num = 0.010207440 Billion write speed all = 0.023180569 MB/sec (with seq num) write speed all = 0.021948557 MB/sec (without seq num) 2024/11/23-03:17:04.564499 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242333: 929434 keys, 38369480 bytes, temperature: kWarm 2024/11/23-03:17:04.564560 140024709862976 EVENT_LOG_v1 {"time_micros": 1732303024564529, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242333, "file_size": 38369480, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3583362754, "largest_seqno": 3754790759, "table_properties": {"data_size": 32251744, "index_size": 1779840, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26024152, "raw_average_key_size": 28, "raw_value_size": 108193604, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929434, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303021, "oldest_key_time": 0, "file_creation_time": 1732303021, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242333, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:17:04.599534 140024709862976 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242334.sst) /dev/shm/ToplingTemp/Topling-392149-nfrCHE 2024/11/23-03:17:04.693040 140024709862976 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242334.sst): first pass time = 0.09's, 244.534'MB/sec 2024/11/23-03:17:04.937904 140024709862976 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242334.sst): old prealloc_size = 147639500, real_size = 6680624 2024/11/23-03:17:04.938041 140024709862976 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242334.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 156557 prefix = 4 raw-key = 0.0025 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0185 GB zip-val = 0.0063 GB avg-val = 118.00 avg-zval = 40.50 2024/11/23-03:17:04.938211 140024709862976 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242334.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.023 GB 2024/11/23-03:17:05.083924 140024709862976 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00/242334.sst): second pass time = 0.24's, 75.631'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.15's, 50.170'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2624.011'MB/sec (index lex order gen) rebuild zvType time = 0.12's, 0.316'MB/sec write SST data time = 0.02's, 332.001'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.193'MB, throughput = 8218.530'MB/sec zip my value time = 0.24's, unzip length = 0.018'GB zip my value throughput = 75.631'MB/sec zip pipeline throughput = 75.631'MB/sec entries = 156557 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 40.50 usrkeys = 156557 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 40.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 156557 bytes = 489248 } seq expand size = 1095899 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0031 GB value = 0.0185 GB dict = 0.19 MB all = 0.0229 GB } __ZipSize{ index = 0.0003 GB value = 0.0063 GB dict = 0.19 MB all = 0.0073 GB } UnZip/Zip{ index = 10.3961 value = 2.9138 dict = 1.00 all = 3.1204 } Zip/UnZip{ index = 0.0962 value = 0.3432 dict = 1.00 all = 0.3205 } ---------------------------- total value len = 1.270203 GB avg = 0.123 KB (by entry num) total key len = 0.290192 GB avg = 0.028 KB total ukey len = 0.207280 GB avg = 0.020 KB total ukey num = 0.010363997 Billion total entry num = 0.010363997 Billion write speed all = 0.023541758 MB/sec (with seq num) write speed all = 0.022290860 MB/sec (without seq num) 2024/11/23-03:17:05.097624 140024709862976 [compaction_job.cc:1948] [default] [JOB 18149] Generated table #242334: 156557 keys, 7325185 bytes, temperature: kWarm 2024/11/23-03:17:05.097681 140024709862976 EVENT_LOG_v1 {"time_micros": 1732303025097651, "cf_name": "default", "job": 18149, "event": "table_file_creation", "file_number": 242334, "file_size": 7325185, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3717273833, "largest_seqno": 3748895427, "table_properties": {"data_size": 6340128, "index_size": 301184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4383596, "raw_average_key_size": 28, "raw_value_size": 18473726, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 156557, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732303024, "oldest_key_time": 0, "file_creation_time": 1732303024, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 242334, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/23-03:17:05.109069 140024709862976 [compaction_job.cc:703] job-18149: subcompact[0], size: 0.419045 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-18149/att-00 [242323,242324,242325,242326,242327,242328,242329,242330,242331,242332,242333,242334] 2024/11/23-03:17:05.201206 140024709862976 (Original Log Time 2024/11/23-03:16:22.016610) EVENT_LOG_v1 {"time_micros": 1732302982016548, "job": 18149, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [242315], "files_L3": [242257, 242258, 242259, 242260, 242261, 242263, 242264, 242265, 242266, 242267, 242270], "score": 0.994968, "input_data_size": 439602628, "oldest_snapshot_seqno": 3754811521} 2024/11/23-03:17:05.201210 140024709862976 (Original Log Time 2024/11/23-03:17:05.200522) [compaction_job.cc:2022] [default] [JOB 18149] Compacted 1@2 + 11@3 files to L3 => 419045436 bytes 2024/11/23-03:17:05.201212 140024709862976 (Original Log Time 2024/11/23-03:17:05.201141) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 10.2 rd, 9.7 wr, level 3, files in(1, 11) out(12 +0 blob) MB in(27.1, 392.1 +0.0 blob) out(399.6 +0.0 blob), read-write-amplify(30.2) write-amplify(14.7) OK, records in: 10439004, records dropped: 75007 output_compression: Snappy 2024/11/23-03:17:05.201217 140024709862976 (Original Log Time 2024/11/23-03:17:05.201186) EVENT_LOG_v1 {"time_micros": 1732303025201165, "cf": "default", "job": 18149, "event": "compaction_finished", "compaction_time_micros": 43092464, "compaction_time_cpu_micros": 18672929, "output_level": 3, "num_output_files": 12, "total_output_size": 419045436, "num_input_records": 10439004, "num_output_records": 10363997, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 281956222, "file_range_sync_nanos": 0, "file_fsync_nanos": 11324, "file_prepare_write_nanos": 29133, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/23-03:17:05.346010 140024709862976 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732302981,"executesMs":43324,"compactionJobId":18149,"attempt":0,"compactionInputRawBytes":1509903956,"compactionInputZipBytes":439602628,"compactionOutputRawBytes":0,"compactionOutputZipBytes":419045436,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-c3ba7519441bab1caad1ac6feb81ebf7-a65d7239e1d4e5f1-00"}}, response = 2024/11/23-03:17:05.346079 140024709862976 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-18149/att-00: olev 3, work 43.325 s, result 4.914 ms, install 0.715 ms, input{raw 1.406 GiB zip 419.238 MiB}