2024/11/24-03:11:09.023205 139957517301312 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00: fork to child time = 0.007763 sec 2024/11/24-03:11:09.023934 139957517301312 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00 2024/11/24-03:11:09.032794 139957517301312 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00 2024/11/24-03:11:09.033688 139957517301312 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-03:11:09.033952 139957517301312 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-03:11:09.087240 139957517301312 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40616/att-00/MANIFEST-206347 2024/11/24-03:11:09.114751 139957517301312 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40616/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 289910, last_sequence is 4411462010, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 289746 2024/11/24-03:11:09.114758 139957517301312 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-03:11:09.114887 139957517301312 [version_set.cc:5721] Creating manifest 289910 2024/11/24-03:11:09.149902 139957517301312 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-03:11:09.169575 139957517301312 [compaction_job.cc:2437] [default] [JOB 40616] Compacting 1@2 + 17@3 files to L3, score 0.79, subcompactions 1 : 1 2024/11/24-03:11:09.169589 139957517301312 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 30424 Base level 2, inputs: [289877(27MB)], [289822(33MB) 289823(35MB) 289824(35MB) 289825(35MB) 289826(33MB) 289827(34MB) 289828(35MB) 289829(35MB) 289830(2792KB) 289804(35MB) 289805(33MB) 289806(33MB) 289807(35MB) 289808(17MB) 289809(33MB) 289810(35MB) 289902(35MB)] 2024/11/24-03:11:09.176329 139957517301312 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-03:11:09.176359 139957517301312 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-03:11:09.176377 139957517301312 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-03:11:09.176382 139957517301312 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-03:11:09.176391 139957517301312 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-03:11:09.865318 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289911.sst) /dev/shm/ToplingTemp/Topling-255729-YogtPf 2024/11/24-03:11:10.555528 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289911.sst): first pass time = 0.69's, 194.470'MB/sec 2024/11/24-03:11:12.027295 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289911.sst): old prealloc_size = 147639500, real_size = 31497792 2024/11/24-03:11:12.027972 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289911.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919541 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0295 GB avg-val = 117.96 avg-zval = 32.09 2024/11/24-03:11:12.028875 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289911.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:12.963986 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289911.sst): second pass time = 1.47's, 73.748'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 37.981'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2640.497'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.327'MB/sec write SST data time = 0.23's, 152.152'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 36180.193'MB/sec zip my value time = 1.47's, unzip length = 0.108'GB zip my value throughput = 73.748'MB/sec zip pipeline throughput = 73.748'MB/sec entries = 919541 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.96 avg-zval = 32.09 usrkeys = 919541 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.96 avg-zval = 32.09 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919541 bytes = 3218416 } seq expand size = 6436787 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.09 MB all = 0.0356 GB } UnZip/Zip{ index = 10.4448 value = 3.6761 dict = 1.00 all = 3.7728 } Zip/UnZip{ index = 0.0957 value = 0.2720 dict = 1.00 all = 0.2651 } ---------------------------- total value len = 2251.440481 GB avg = 0.200 KB (by entry num) total key len = 306.908710 GB avg = 0.027 KB total ukey len = 237.373118 GB avg = 0.015 KB total ukey num = 15.739580704 Billion total entry num = 11.280894652 Billion write speed all = 38.903893547 MB/sec (with seq num) write speed all = 37.531537572 MB/sec (without seq num) 2024/11/24-03:11:13.013335 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289911: 919541 keys, 35575300 bytes, temperature: kWarm 2024/11/24-03:11:13.013468 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389073013386, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289911, "file_size": 35575300, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4259662629, "largest_seqno": 4411292330, "table_properties": {"data_size": 29506928, "index_size": 1760768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25747148, "raw_average_key_size": 28, "raw_value_size": 108470582, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 919541, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389069, "oldest_key_time": 0, "file_creation_time": 1732389069, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289911, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:13.065014 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289912.sst) /dev/shm/ToplingTemp/Topling-255729-Bz2xJc 2024/11/24-03:11:14.032494 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289912.sst): first pass time = 0.97's, 138.740'MB/sec 2024/11/24-03:11:15.341455 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289912.sst): old prealloc_size = 147639500, real_size = 33005024 2024/11/24-03:11:15.342020 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289912.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926402 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0310 GB avg-val = 116.88 avg-zval = 33.47 2024/11/24-03:11:15.342778 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289912.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:16.193166 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289912.sst): second pass time = 1.31's, 82.787'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.563'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3229.747'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.319'MB/sec write SST data time = 0.13's, 294.617'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34596.816'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.787'MB/sec zip pipeline throughput = 82.787'MB/sec entries = 926402 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.88 avg-zval = 33.47 usrkeys = 926402 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.88 avg-zval = 33.47 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926402 bytes = 3242416 } seq expand size = 6484814 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5143 value = 3.4916 dict = 1.00 all = 3.6176 } Zip/UnZip{ index = 0.0951 value = 0.2864 dict = 1.00 all = 0.2764 } ---------------------------- total value len = 2251.555244 GB avg = 0.200 KB (by entry num) total key len = 306.934649 GB avg = 0.027 KB total ukey len = 237.391646 GB avg = 0.015 KB total ukey num = 15.740507106 Billion total entry num = 11.281821054 Billion write speed all = 38.904122754 MB/sec (with seq num) write speed all = 37.531721471 MB/sec (without seq num) 2024/11/24-03:11:16.241848 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289912: 926402 keys, 37101726 bytes, temperature: kWarm 2024/11/24-03:11:16.242051 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389076241884, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289912, "file_size": 37101726, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4259662645, "largest_seqno": 4411292605, "table_properties": {"data_size": 31011088, "index_size": 1762176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25939256, "raw_average_key_size": 28, "raw_value_size": 108278480, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926402, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389073, "oldest_key_time": 0, "file_creation_time": 1732389073, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289912, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:16.286036 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289913.sst) /dev/shm/ToplingTemp/Topling-255729-bK1D6c 2024/11/24-03:11:17.151558 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289913.sst): first pass time = 0.87's, 155.080'MB/sec 2024/11/24-03:11:18.638514 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289913.sst): old prealloc_size = 147639500, real_size = 33465264 2024/11/24-03:11:18.639090 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289913.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926696 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.83 avg-zval = 33.95 2024/11/24-03:11:18.639847 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289913.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:19.498802 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289913.sst): second pass time = 1.49's, 72.862'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.665'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3191.604'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.315'MB/sec write SST data time = 0.13's, 299.676'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34696.786'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.862'MB/sec zip pipeline throughput = 72.862'MB/sec entries = 926696 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.83 avg-zval = 33.95 usrkeys = 926696 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.83 avg-zval = 33.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926696 bytes = 3243456 } seq expand size = 6486872 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.4425 value = 3.4417 dict = 1.00 all = 3.5731 } Zip/UnZip{ index = 0.0958 value = 0.2906 dict = 1.00 all = 0.2799 } ---------------------------- total value len = 2251.670001 GB avg = 0.200 KB (by entry num) total key len = 306.960596 GB avg = 0.027 KB total ukey len = 237.410180 GB avg = 0.015 KB total ukey num = 15.741433802 Billion total entry num = 11.282747750 Billion write speed all = 38.904306762 MB/sec (with seq num) write speed all = 37.531861736 MB/sec (without seq num) 2024/11/24-03:11:19.546544 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289913: 926696 keys, 37563458 bytes, temperature: kWarm 2024/11/24-03:11:19.546612 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389079546578, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289913, "file_size": 37563458, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178503, "largest_seqno": 4411292347, "table_properties": {"data_size": 31458528, "index_size": 1774848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25947488, "raw_average_key_size": 28, "raw_value_size": 108270268, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926696, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389076, "oldest_key_time": 0, "file_creation_time": 1732389076, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289913, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:19.583182 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289914.sst) /dev/shm/ToplingTemp/Topling-255729-kKW9Vb 2024/11/24-03:11:20.532664 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289914.sst): first pass time = 0.95's, 141.365'MB/sec 2024/11/24-03:11:21.975318 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289914.sst): old prealloc_size = 147639500, real_size = 32580480 2024/11/24-03:11:21.975839 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289914.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926635 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0306 GB avg-val = 116.84 avg-zval = 33.01 2024/11/24-03:11:21.976626 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289914.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:22.811939 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289914.sst): second pass time = 1.44's, 75.121'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.843'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3543.997'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.325'MB/sec write SST data time = 0.12's, 296.322'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34579.293'MB/sec zip my value time = 1.44's, unzip length = 0.108'GB zip my value throughput = 75.121'MB/sec zip pipeline throughput = 75.121'MB/sec entries = 926635 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 33.01 usrkeys = 926635 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.84 avg-zval = 33.01 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926635 bytes = 3243232 } seq expand size = 6486445 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5127 value = 3.5399 dict = 1.00 all = 3.6593 } Zip/UnZip{ index = 0.0951 value = 0.2825 dict = 1.00 all = 0.2733 } ---------------------------- total value len = 2251.784759 GB avg = 0.200 KB (by entry num) total key len = 306.986542 GB avg = 0.027 KB total ukey len = 237.428713 GB avg = 0.015 KB total ukey num = 15.742360437 Billion total entry num = 11.283674385 Billion write speed all = 38.904486310 MB/sec (with seq num) write speed all = 37.531997709 MB/sec (without seq num) 2024/11/24-03:11:22.858842 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289914: 926635 keys, 36678334 bytes, temperature: kWarm 2024/11/24-03:11:22.858913 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389082858878, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289914, "file_size": 36678334, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178518, "largest_seqno": 4411292363, "table_properties": {"data_size": 30585776, "index_size": 1762880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945780, "raw_average_key_size": 28, "raw_value_size": 108272046, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926635, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389079, "oldest_key_time": 0, "file_creation_time": 1732389079, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289914, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:22.894401 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289915.sst) /dev/shm/ToplingTemp/Topling-255729-1gkuIc 2024/11/24-03:11:23.811469 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289915.sst): first pass time = 0.92's, 146.361'MB/sec 2024/11/24-03:11:25.243123 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289915.sst): old prealloc_size = 147639500, real_size = 30986384 2024/11/24-03:11:25.243633 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289915.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924649 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0290 GB avg-val = 117.16 avg-zval = 31.35 2024/11/24-03:11:25.244387 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289915.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:26.053359 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289915.sst): second pass time = 1.43's, 75.707'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.151'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3636.099'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.338'MB/sec write SST data time = 0.13's, 279.062'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 29134.052'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.707'MB/sec zip pipeline throughput = 82.183'MB/sec entries = 924649 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.16 avg-zval = 31.35 usrkeys = 924649 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.16 avg-zval = 31.35 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924649 bytes = 3120704 } seq expand size = 6472543 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0290 GB dict = 1.08 MB all = 0.0350 GB } UnZip/Zip{ index = 10.4443 value = 3.7374 dict = 1.00 all = 3.8390 } Zip/UnZip{ index = 0.0957 value = 0.2676 dict = 1.00 all = 0.2605 } ---------------------------- total value len = 2251.899560 GB avg = 0.200 KB (by entry num) total key len = 307.012432 GB avg = 0.027 KB total ukey len = 237.447206 GB avg = 0.015 KB total ukey num = 15.743285086 Billion total entry num = 11.284599034 Billion write speed all = 38.904708047 MB/sec (with seq num) write speed all = 37.532174619 MB/sec (without seq num) 2024/11/24-03:11:26.098712 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289915: 924649 keys, 34961344 bytes, temperature: kWarm 2024/11/24-03:11:26.098775 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389086098739, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289915, "file_size": 34961344, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4290314967, "largest_seqno": 4411292380, "table_properties": {"data_size": 28984384, "index_size": 1770624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25890172, "raw_average_key_size": 28, "raw_value_size": 108327562, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924649, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389082, "oldest_key_time": 0, "file_creation_time": 1732389082, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289915, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:26.134776 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289916.sst) /dev/shm/ToplingTemp/Topling-255729-01jLfc 2024/11/24-03:11:26.957986 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289916.sst): first pass time = 0.82's, 163.051'MB/sec 2024/11/24-03:11:28.481083 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289916.sst): old prealloc_size = 147639500, real_size = 32170640 2024/11/24-03:11:28.481589 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289916.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921321 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0302 GB avg-val = 117.68 avg-zval = 32.77 2024/11/24-03:11:28.482352 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289916.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:29.301580 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289916.sst): second pass time = 1.52's, 71.240'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 44.186'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3606.776'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.332'MB/sec write SST data time = 0.13's, 288.775'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 37419.455'MB/sec zip my value time = 1.52's, unzip length = 0.108'GB zip my value throughput = 71.240'MB/sec zip pipeline throughput = 102.458'MB/sec entries = 921321 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.68 avg-zval = 32.77 usrkeys = 921321 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.68 avg-zval = 32.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921321 bytes = 3224640 } seq expand size = 6449247 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.5124 value = 3.5916 dict = 1.00 all = 3.7021 } Zip/UnZip{ index = 0.0951 value = 0.2784 dict = 1.00 all = 0.2701 } ---------------------------- total value len = 2252.014430 GB avg = 0.200 KB (by entry num) total key len = 307.038229 GB avg = 0.027 KB total ukey len = 237.465632 GB avg = 0.015 KB total ukey num = 15.744206407 Billion total entry num = 11.285520355 Billion write speed all = 38.904925383 MB/sec (with seq num) write speed all = 37.532347681 MB/sec (without seq num) 2024/11/24-03:11:29.350712 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289916: 921321 keys, 36254398 bytes, temperature: kWarm 2024/11/24-03:11:29.350781 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389089350745, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289916, "file_size": 36254398, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178530, "largest_seqno": 4411293314, "table_properties": {"data_size": 30187264, "index_size": 1752832, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25796988, "raw_average_key_size": 28, "raw_value_size": 108420798, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921321, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389086, "oldest_key_time": 0, "file_creation_time": 1732389086, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289916, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:29.387671 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289917.sst) /dev/shm/ToplingTemp/Topling-255729-jPcK1f 2024/11/24-03:11:30.289138 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289917.sst): first pass time = 0.90's, 148.895'MB/sec 2024/11/24-03:11:31.856663 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289917.sst): old prealloc_size = 147639500, real_size = 33279056 2024/11/24-03:11:31.857204 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289917.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926607 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 116.85 avg-zval = 33.76 2024/11/24-03:11:31.857943 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289917.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:32.711769 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289917.sst): second pass time = 1.57's, 69.120'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.711'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3371.198'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.12's, 299.604'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 37439.407'MB/sec zip my value time = 1.57's, unzip length = 0.108'GB zip my value throughput = 69.120'MB/sec zip pipeline throughput = 109.685'MB/sec entries = 926607 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 33.76 usrkeys = 926607 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 33.76 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926607 bytes = 3243136 } seq expand size = 6486249 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5143 value = 3.4609 dict = 1.00 all = 3.5910 } Zip/UnZip{ index = 0.0951 value = 0.2889 dict = 1.00 all = 0.2785 } ---------------------------- total value len = 2252.129189 GB avg = 0.200 KB (by entry num) total key len = 307.064174 GB avg = 0.027 KB total ukey len = 237.484164 GB avg = 0.015 KB total ukey num = 15.745133014 Billion total entry num = 11.286446962 Billion write speed all = 38.905047469 MB/sec (with seq num) write speed all = 37.532428233 MB/sec (without seq num) 2024/11/24-03:11:32.758193 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289917: 926607 keys, 37376452 bytes, temperature: kWarm 2024/11/24-03:11:32.758259 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389092758225, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289917, "file_size": 37376452, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178548, "largest_seqno": 4411292397, "table_properties": {"data_size": 31284672, "index_size": 1762560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944996, "raw_average_key_size": 28, "raw_value_size": 108272778, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926607, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389089, "oldest_key_time": 0, "file_creation_time": 1732389089, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289917, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:32.793122 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289918.sst) /dev/shm/ToplingTemp/Topling-255729-2QSgdd 2024/11/24-03:11:33.689878 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289918.sst): first pass time = 0.90's, 149.676'MB/sec 2024/11/24-03:11:35.304793 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289918.sst): old prealloc_size = 147639500, real_size = 33349136 2024/11/24-03:11:35.305327 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289918.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926547 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.86 avg-zval = 33.84 2024/11/24-03:11:35.306079 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289918.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:36.156351 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289918.sst): second pass time = 1.61's, 67.078'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.976'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3430.731'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.319'MB/sec write SST data time = 0.13's, 299.231'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33586.224'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.078'MB/sec zip pipeline throughput = 119.633'MB/sec entries = 926547 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 33.84 usrkeys = 926547 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 33.84 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926547 bytes = 3242928 } seq expand size = 6485829 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5133 value = 3.4532 dict = 1.00 all = 3.5842 } Zip/UnZip{ index = 0.0951 value = 0.2896 dict = 1.00 all = 0.2790 } ---------------------------- total value len = 2252.243949 GB avg = 0.200 KB (by entry num) total key len = 307.090118 GB avg = 0.027 KB total ukey len = 237.502695 GB avg = 0.015 KB total ukey num = 15.746059561 Billion total entry num = 11.287373509 Billion write speed all = 38.905149199 MB/sec (with seq num) write speed all = 37.532489158 MB/sec (without seq num) 2024/11/24-03:11:36.203615 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289918: 926547 keys, 37446760 bytes, temperature: kWarm 2024/11/24-03:11:36.203681 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389096203646, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289918, "file_size": 37446760, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178559, "largest_seqno": 4411292417, "table_properties": {"data_size": 31354688, "index_size": 1762624, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25943316, "raw_average_key_size": 28, "raw_value_size": 108274526, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926547, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389092, "oldest_key_time": 0, "file_creation_time": 1732389092, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289918, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:36.242375 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289919.sst) /dev/shm/ToplingTemp/Topling-255729-WERKgg 2024/11/24-03:11:37.234671 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289919.sst): first pass time = 0.99's, 135.265'MB/sec 2024/11/24-03:11:38.666810 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289919.sst): old prealloc_size = 147639500, real_size = 32177360 2024/11/24-03:11:38.667392 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289919.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926617 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0302 GB avg-val = 116.85 avg-zval = 32.57 2024/11/24-03:11:38.668747 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289919.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:39.502512 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289919.sst): second pass time = 1.43's, 75.642'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.408'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3139.125'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.13's, 281.853'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 27420.763'MB/sec zip my value time = 1.43's, unzip length = 0.108'GB zip my value throughput = 75.642'MB/sec zip pipeline throughput = 75.642'MB/sec entries = 926617 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 32.57 usrkeys = 926617 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 32.57 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926617 bytes = 3243168 } seq expand size = 6486319 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0302 GB dict = 1.08 MB all = 0.0363 GB } UnZip/Zip{ index = 10.5148 value = 3.5872 dict = 1.00 all = 3.7000 } Zip/UnZip{ index = 0.0951 value = 0.2788 dict = 1.00 all = 0.2703 } ---------------------------- total value len = 2252.358708 GB avg = 0.200 KB (by entry num) total key len = 307.116063 GB avg = 0.027 KB total ukey len = 237.521228 GB avg = 0.015 KB total ukey num = 15.746986178 Billion total entry num = 11.288300126 Billion write speed all = 38.905309128 MB/sec (with seq num) write speed all = 37.532606225 MB/sec (without seq num) 2024/11/24-03:11:39.549701 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289919: 926617 keys, 36275274 bytes, temperature: kWarm 2024/11/24-03:11:39.549769 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389099549734, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289919, "file_size": 36275274, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178575, "largest_seqno": 4411293563, "table_properties": {"data_size": 30183040, "index_size": 1762496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25945276, "raw_average_key_size": 28, "raw_value_size": 108272506, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926617, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389096, "oldest_key_time": 0, "file_creation_time": 1732389096, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289919, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:39.588500 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289920.sst) /dev/shm/ToplingTemp/Topling-255729-QrW4Ff 2024/11/24-03:11:40.574017 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289920.sst): first pass time = 0.99's, 136.196'MB/sec 2024/11/24-03:11:42.030160 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289920.sst): old prealloc_size = 147639500, real_size = 31085136 2024/11/24-03:11:42.030665 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289920.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923302 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0291 GB avg-val = 117.37 avg-zval = 31.50 2024/11/24-03:11:42.031457 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289920.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:42.841200 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289920.sst): second pass time = 1.46's, 74.458'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.226'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3676.154'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.337'MB/sec write SST data time = 0.13's, 278.399'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 27765.750'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.458'MB/sec zip pipeline throughput = 74.458'MB/sec entries = 923302 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.37 avg-zval = 31.50 usrkeys = 923302 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.37 avg-zval = 31.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923302 bytes = 3116160 } seq expand size = 6463114 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0291 GB dict = 1.08 MB all = 0.0351 GB } UnZip/Zip{ index = 10.4431 value = 3.7257 dict = 1.00 all = 3.8285 } Zip/UnZip{ index = 0.0958 value = 0.2684 dict = 1.00 all = 0.2612 } ---------------------------- total value len = 2252.473536 GB avg = 0.200 KB (by entry num) total key len = 307.141915 GB avg = 0.027 KB total ukey len = 237.539694 GB avg = 0.015 KB total ukey num = 15.747909480 Billion total entry num = 11.289223428 Billion write speed all = 38.905473107 MB/sec (with seq num) write speed all = 37.532727593 MB/sec (without seq num) 2024/11/24-03:11:42.885344 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289920: 923302 keys, 35057372 bytes, temperature: kWarm 2024/11/24-03:11:42.885416 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389102885379, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289920, "file_size": 35057372, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4282173089, "largest_seqno": 4411292452, "table_properties": {"data_size": 29085888, "index_size": 1768256, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25852456, "raw_average_key_size": 28, "raw_value_size": 108365312, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923302, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389099, "oldest_key_time": 0, "file_creation_time": 1732389099, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289920, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:42.927201 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289921.sst) /dev/shm/ToplingTemp/Topling-255729-EJesMe 2024/11/24-03:11:43.941788 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289921.sst): first pass time = 1.01's, 132.293'MB/sec 2024/11/24-03:11:45.397255 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289921.sst): old prealloc_size = 147639500, real_size = 32591936 2024/11/24-03:11:45.397801 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289921.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922538 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0306 GB avg-val = 117.49 avg-zval = 33.18 2024/11/24-03:11:45.399810 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289921.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:46.232559 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289921.sst): second pass time = 1.45's, 74.527'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.912'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3406.335'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.13's, 289.587'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 36514.498'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.527'MB/sec zip pipeline throughput = 74.527'MB/sec entries = 922538 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.49 avg-zval = 33.18 usrkeys = 922538 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.49 avg-zval = 33.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922538 bytes = 3228896 } seq expand size = 6457766 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5136 value = 3.5413 dict = 1.00 all = 3.6592 } Zip/UnZip{ index = 0.0951 value = 0.2824 dict = 1.00 all = 0.2733 } ---------------------------- total value len = 2252.588381 GB avg = 0.200 KB (by entry num) total key len = 307.167746 GB avg = 0.027 KB total ukey len = 237.558144 GB avg = 0.015 KB total ukey num = 15.748832018 Billion total entry num = 11.290145966 Billion write speed all = 38.905605841 MB/sec (with seq num) write speed all = 37.532818913 MB/sec (without seq num) 2024/11/24-03:11:46.282393 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289921: 922538 keys, 36679068 bytes, temperature: kWarm 2024/11/24-03:11:46.282462 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389106282428, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289921, "file_size": 36679068, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178589, "largest_seqno": 4411290420, "table_properties": {"data_size": 30606192, "index_size": 1754944, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25831064, "raw_average_key_size": 28, "raw_value_size": 108386728, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922538, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389102, "oldest_key_time": 0, "file_creation_time": 1732389102, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289921, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:46.326585 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289922.sst) /dev/shm/ToplingTemp/Topling-255729-0ltVhe 2024/11/24-03:11:47.332323 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289922.sst): first pass time = 1.01's, 133.457'MB/sec 2024/11/24-03:11:48.862300 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289922.sst): old prealloc_size = 147639500, real_size = 33502688 2024/11/24-03:11:48.862864 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289922.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926591 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.85 avg-zval = 33.99 2024/11/24-03:11:48.864892 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289922.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:49.727236 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289922.sst): second pass time = 1.53's, 70.803'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.473'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3260.514'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.13's, 299.018'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 35454.056'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.803'MB/sec zip pipeline throughput = 70.803'MB/sec entries = 926591 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.85 avg-zval = 33.99 usrkeys = 926591 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.85 avg-zval = 33.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926591 bytes = 3243088 } seq expand size = 6486137 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.4432 value = 3.4376 dict = 1.00 all = 3.5696 } Zip/UnZip{ index = 0.0958 value = 0.2909 dict = 1.00 all = 0.2801 } ---------------------------- total value len = 2252.703140 GB avg = 0.200 KB (by entry num) total key len = 307.193691 GB avg = 0.027 KB total ukey len = 237.576676 GB avg = 0.015 KB total ukey num = 15.749758609 Billion total entry num = 11.291072557 Billion write speed all = 38.905677899 MB/sec (with seq num) write speed all = 37.532851223 MB/sec (without seq num) 2024/11/24-03:11:49.774414 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289922: 926591 keys, 37600372 bytes, temperature: kWarm 2024/11/24-03:11:49.774481 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389109774447, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289922, "file_size": 37600372, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178599, "largest_seqno": 4411293523, "table_properties": {"data_size": 31496336, "index_size": 1774528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944548, "raw_average_key_size": 28, "raw_value_size": 108273202, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926591, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389106, "oldest_key_time": 0, "file_creation_time": 1732389106, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289922, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:49.812769 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289923.sst) /dev/shm/ToplingTemp/Topling-255729-VuXkOf 2024/11/24-03:11:50.679390 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289923.sst): first pass time = 0.87's, 101.261'MB/sec 2024/11/24-03:11:51.649112 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289923.sst): old prealloc_size = 147639500, real_size = 21119680 2024/11/24-03:11:51.649506 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289923.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 608450 prefix = 4 raw-key = 0.0097 GB zip-key = 0.0012 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0707 GB zip-val = 0.0198 GB avg-val = 116.22 avg-zval = 32.54 2024/11/24-03:11:51.650009 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289923.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0015 GB, waited 0.000 sec, Key+Value = 0.088 GB 2024/11/24-03:11:52.213797 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289923.sst): second pass time = 0.97's, 72.977'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.56's, 41.897'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3154.623'MB/sec (index lex order gen) rebuild zvType time = 0.48's, 0.316'MB/sec write SST data time = 0.08's, 283.797'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.710'MB, throughput = 24092.150'MB/sec zip my value time = 0.97's, unzip length = 0.071'GB zip my value throughput = 72.977'MB/sec zip pipeline throughput = 72.977'MB/sec entries = 608450 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.22 avg-zval = 32.54 usrkeys = 608450 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.22 avg-zval = 32.54 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 608450 bytes = 1977472 } seq expand size = 4259150 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0122 GB value = 0.0707 GB dict = 0.71 MB all = 0.0878 GB } __ZipSize{ index = 0.0012 GB value = 0.0198 GB dict = 0.71 MB all = 0.0237 GB } UnZip/Zip{ index = 10.4410 value = 3.5711 dict = 1.00 all = 3.7092 } Zip/UnZip{ index = 0.0958 value = 0.2800 dict = 1.00 all = 0.2696 } ---------------------------- total value len = 2252.778114 GB avg = 0.200 KB (by entry num) total key len = 307.210728 GB avg = 0.027 KB total ukey len = 237.588845 GB avg = 0.015 KB total ukey num = 15.750367059 Billion total entry num = 11.291681007 Billion write speed all = 38.905605990 MB/sec (with seq num) write speed all = 37.532757218 MB/sec (without seq num) 2024/11/24-03:11:52.247346 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289923: 608450 keys, 23657673 bytes, temperature: kWarm 2024/11/24-03:11:52.247418 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389112247382, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289923, "file_size": 23657673, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4355345807, "largest_seqno": 4411293213, "table_properties": {"data_size": 19801904, "index_size": 1165504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 17036600, "raw_average_key_size": 28, "raw_value_size": 70714252, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 608450, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389109, "oldest_key_time": 0, "file_creation_time": 1732389109, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289923, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:52.281465 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289924.sst) /dev/shm/ToplingTemp/Topling-255729-ztDWvg 2024/11/24-03:11:52.824574 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289924.sst): first pass time = 0.54's, 247.146'MB/sec 2024/11/24-03:11:54.186410 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289924.sst): old prealloc_size = 147639500, real_size = 31371696 2024/11/24-03:11:54.186963 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289924.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919835 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0294 GB avg-val = 117.92 avg-zval = 31.94 2024/11/24-03:11:54.187745 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289924.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:55.108538 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289924.sst): second pass time = 1.36's, 79.694'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 38.445'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3240.038'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 283.298'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 34320.762'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.694'MB/sec zip pipeline throughput = 79.694'MB/sec entries = 919835 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.92 avg-zval = 31.94 usrkeys = 919835 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.92 avg-zval = 31.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919835 bytes = 3219440 } seq expand size = 6438845 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.08 MB all = 0.0354 GB } UnZip/Zip{ index = 10.4458 value = 3.6917 dict = 1.00 all = 3.7861 } Zip/UnZip{ index = 0.0957 value = 0.2709 dict = 1.00 all = 0.2641 } ---------------------------- total value len = 2252.893015 GB avg = 0.200 KB (by entry num) total key len = 307.236483 GB avg = 0.027 KB total ukey len = 237.607242 GB avg = 0.015 KB total ukey num = 15.751286894 Billion total entry num = 11.292600842 Billion write speed all = 38.906032038 MB/sec (with seq num) write speed all = 37.533131829 MB/sec (without seq num) 2024/11/24-03:11:55.151727 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289924: 919835 keys, 35449878 bytes, temperature: kWarm 2024/11/24-03:11:55.151794 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389115151760, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289924, "file_size": 35449878, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178645, "largest_seqno": 4411293780, "table_properties": {"data_size": 29380384, "index_size": 1761152, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25755380, "raw_average_key_size": 28, "raw_value_size": 108462374, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 919835, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389112, "oldest_key_time": 0, "file_creation_time": 1732389112, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289924, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:55.187305 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289925.sst) /dev/shm/ToplingTemp/Topling-255729-BRLZcd 2024/11/24-03:11:56.023319 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289925.sst): first pass time = 0.84's, 160.552'MB/sec 2024/11/24-03:11:57.391672 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289925.sst): old prealloc_size = 147639500, real_size = 33102064 2024/11/24-03:11:57.392262 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289925.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926108 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 116.93 avg-zval = 33.58 2024/11/24-03:11:57.393033 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289925.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:11:58.343886 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289925.sst): second pass time = 1.37's, 79.183'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.066'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3122.053'MB/sec (index lex order gen) rebuild zvType time = 0.83's, 0.280'MB/sec write SST data time = 0.13's, 295.112'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34249.199'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.183'MB/sec zip pipeline throughput = 79.183'MB/sec entries = 926108 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.93 avg-zval = 33.58 usrkeys = 926108 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.93 avg-zval = 33.58 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926108 bytes = 3241392 } seq expand size = 6482756 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.4423 value = 3.4823 dict = 1.00 all = 3.6082 } Zip/UnZip{ index = 0.0958 value = 0.2872 dict = 1.00 all = 0.2771 } ---------------------------- total value len = 2253.007784 GB avg = 0.199 KB (by entry num) total key len = 307.262414 GB avg = 0.027 KB total ukey len = 237.625764 GB avg = 0.015 KB total ukey num = 15.752213002 Billion total entry num = 11.293526950 Billion write speed all = 38.906257334 MB/sec (with seq num) write speed all = 37.533312038 MB/sec (without seq num) 2024/11/24-03:11:58.388807 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289925: 926108 keys, 37198094 bytes, temperature: kWarm 2024/11/24-03:11:58.388880 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389118388842, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289925, "file_size": 37198094, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178655, "largest_seqno": 4411293407, "table_properties": {"data_size": 31096608, "index_size": 1773760, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25931024, "raw_average_key_size": 28, "raw_value_size": 108286732, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926108, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389115, "oldest_key_time": 0, "file_creation_time": 1732389115, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289925, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:11:58.426224 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289926.sst) /dev/shm/ToplingTemp/Topling-255729-TAx58d 2024/11/24-03:11:59.473395 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289926.sst): first pass time = 1.05's, 128.177'MB/sec 2024/11/24-03:12:00.876641 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289926.sst): old prealloc_size = 147639500, real_size = 33500368 2024/11/24-03:12:00.877369 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289926.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926392 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.88 avg-zval = 34.01 2024/11/24-03:12:00.878856 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289926.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:12:01.832721 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289926.sst): second pass time = 1.40's, 77.213'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 39.326'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2487.982'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.290'MB/sec write SST data time = 0.16's, 241.132'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 25498.242'MB/sec zip my value time = 1.40's, unzip length = 0.108'GB zip my value throughput = 77.213'MB/sec zip pipeline throughput = 77.213'MB/sec entries = 926392 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.88 avg-zval = 34.01 usrkeys = 926392 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.88 avg-zval = 34.01 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926392 bytes = 3242384 } seq expand size = 6484744 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5138 value = 3.4367 dict = 1.00 all = 3.5699 } Zip/UnZip{ index = 0.0951 value = 0.2910 dict = 1.00 all = 0.2801 } ---------------------------- total value len = 2253.122548 GB avg = 0.199 KB (by entry num) total key len = 307.288353 GB avg = 0.027 KB total ukey len = 237.644292 GB avg = 0.015 KB total ukey num = 15.753139394 Billion total entry num = 11.294453342 Billion write speed all = 38.906332783 MB/sec (with seq num) write speed all = 37.533347657 MB/sec (without seq num) 2024/11/24-03:12:01.878910 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289926: 926392 keys, 37597598 bytes, temperature: kWarm 2024/11/24-03:12:01.878992 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389121878946, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289926, "file_size": 37597598, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178675, "largest_seqno": 4411294044, "table_properties": {"data_size": 31506368, "index_size": 1762240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25938976, "raw_average_key_size": 28, "raw_value_size": 108278816, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926392, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389118, "oldest_key_time": 0, "file_creation_time": 1732389118, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289926, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:12:01.923883 139957517301312 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289927.sst) /dev/shm/ToplingTemp/Topling-255729-IWHDwe 2024/11/24-03:12:01.941289 139957517301312 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289927.sst): first pass time = 0.02's, 172.428'MB/sec 2024/11/24-03:12:01.982527 139957517301312 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289927.sst): old prealloc_size = 147639500, real_size = 992144 2024/11/24-03:12:01.982596 139957517301312 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289927.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 20496 prefix = 4 raw-key = 0.0003 GB zip-key = 0.0000 GB avg-key = 16.00 avg-zkey = 1.99 raw-val = 0.0024 GB zip-val = 0.0009 GB avg-val = 118.00 avg-zval = 46.16 2024/11/24-03:12:01.982656 139957517301312 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289927.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0000 GB, waited 0.000 sec, Key+Value = 0.003 GB 2024/11/24-03:12:02.002775 139957517301312 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00/289927.sst): second pass time = 0.04's, 59.661'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.02's, 52.820'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 873.575'MB/sec (index lex order gen) rebuild zvType time = 0.01's, 0.366'MB/sec write SST data time = 0.01's, 172.586'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.024'MB, throughput = 817.358'MB/sec zip my value time = 0.04's, unzip length = 0.002'GB zip my value throughput = 59.661'MB/sec zip pipeline throughput = 59.661'MB/sec entries = 20496 avg-key = 28.00 avg-zkey = 1.99 avg-val = 118.00 avg-zval = 46.16 usrkeys = 20496 avg-key = 20.00 avg-zkey = 1.99 avg-val = 125.00 avg-zval = 46.16 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 20496 bytes = 56384 } seq expand size = 143472 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0004 GB value = 0.0024 GB dict = 0.02 MB all = 0.0030 GB } __ZipSize{ index = 0.0000 GB value = 0.0009 GB dict = 0.02 MB all = 0.0011 GB } UnZip/Zip{ index = 10.0549 value = 2.5565 dict = 1.00 all = 2.8000 } Zip/UnZip{ index = 0.0995 value = 0.3912 dict = 1.00 all = 0.3571 } ---------------------------- total value len = 2253.125110 GB avg = 0.199 KB (by entry num) total key len = 307.288927 GB avg = 0.027 KB total ukey len = 237.644702 GB avg = 0.015 KB total ukey num = 15.753159890 Billion total entry num = 11.294473838 Billion write speed all = 38.906279892 MB/sec (with seq num) write speed all = 37.533295823 MB/sec (without seq num) 2024/11/24-03:12:02.015602 139957517301312 [compaction_job.cc:1948] [default] [JOB 40616] Generated table #289927: 20496 keys, 1068705 bytes, temperature: kWarm 2024/11/24-03:12:02.015665 139957517301312 EVENT_LOG_v1 {"time_micros": 1732389122015634, "cf_name": "default", "job": 40616, "event": "table_file_creation", "file_number": 289927, "file_size": 1068705, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4336488065, "largest_seqno": 4339842390, "table_properties": {"data_size": 946048, "index_size": 40768, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 573888, "raw_average_key_size": 28, "raw_value_size": 2418528, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 20496, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732389121, "oldest_key_time": 0, "file_creation_time": 1732389121, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289927, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:12:02.024689 139957517301312 [compaction_job.cc:703] job-40616: subcompact[0], size: 0.573542 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40616/att-00 [289911,289912,289913,289914,289915,289916,289917,289918,289919,289920,289921,289922,289923,289924,289925,289926,289927] 2024/11/24-03:12:02.124441 139957517301312 (Original Log Time 2024/11/24-03:11:09.169639) EVENT_LOG_v1 {"time_micros": 1732389069169600, "job": 40616, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [289877], "files_L3": [289822, 289823, 289824, 289825, 289826, 289827, 289828, 289829, 289830, 289804, 289805, 289806, 289807, 289808, 289809, 289810, 289902], "score": 0.792268, "input_data_size": 596607841, "oldest_snapshot_seqno": 4411461847} 2024/11/24-03:12:02.124446 139957517301312 (Original Log Time 2024/11/24-03:12:02.122906) [compaction_job.cc:2022] [default] [JOB 40616] Compacted 1@2 + 17@3 files to L3 => 573541806 bytes 2024/11/24-03:12:02.124447 139957517301312 (Original Log Time 2024/11/24-03:12:02.124370) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 11.3 rd, 10.9 wr, level 3, files in(1, 17) out(17 +0 blob) MB in(27.1, 541.8 +0.0 blob) out(547.0 +0.0 blob), read-write-amplify(41.1) write-amplify(20.2) OK, records in: 14608275, records dropped: 109548 output_compression: Snappy 2024/11/24-03:12:02.124453 139957517301312 (Original Log Time 2024/11/24-03:12:02.124418) EVENT_LOG_v1 {"time_micros": 1732389122124396, "cf": "default", "job": 40616, "event": "compaction_finished", "compaction_time_micros": 52855060, "compaction_time_cpu_micros": 22427716, "output_level": 3, "num_output_files": 17, "total_output_size": 573541806, "num_input_records": 14608275, "num_output_records": 14498727, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 404147939, "file_range_sync_nanos": 0, "file_fsync_nanos": 13362, "file_prepare_write_nanos": 42453, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/24-03:12:02.240633 139957517301312 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732389068,"executesMs":53091,"compactionJobId":40616,"attempt":0,"compactionInputRawBytes":2119565662,"compactionInputZipBytes":596607841,"compactionOutputRawBytes":0,"compactionOutputZipBytes":573541806,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-63b4b8b1926956cd5784d1bd86fd0684-068a39a992cd74a1-00"}}, response = 2024/11/24-03:12:02.240713 139957517301312 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40616/att-00: olev 3, work 53.092 s, result 7.672 ms, install 1.568 ms, input{raw 1.974 GiB zip 568.970 MiB}