2024/11/22-21:29:53.047184 140024818968128 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00: fork to child time = 0.009073 sec 2024/11/22-21:29:53.047997 140024818968128 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00 2024/11/22-21:29:53.054752 140024818968128 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00 2024/11/22-21:29:53.055880 140024818968128 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-21:29:53.056156 140024818968128 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-21:29:53.102482 140024818968128 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11556/att-00/MANIFEST-206347 2024/11/22-21:29:53.140556 140024818968128 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11556/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 228812, last_sequence is 3576294968, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 228600 2024/11/22-21:29:53.140564 140024818968128 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-21:29:53.140662 140024818968128 [version_set.cc:5721] Creating manifest 228812 2024/11/22-21:29:53.172982 140024818968128 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-21:29:53.193317 140024818968128 [compaction_job.cc:2437] [default] [JOB 11556] Compacting 1@2 + 14@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/22-21:29:53.193336 140024818968128 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 8704 Base level 2, inputs: [228751(26MB)], [228809(33MB) 228810(6898KB) 228708(33MB) 228709(36MB) 228710(33MB) 228711(34MB) 228712(36MB) 228713(35MB) 228714(36MB) 228715(10MB) 228679(36MB) 228680(35MB) 228681(36MB) 228789(36MB)] 2024/11/22-21:29:53.200816 140024818968128 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:29:53.200848 140024818968128 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:29:53.200868 140024818968128 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-21:29:53.200873 140024818968128 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-21:29:53.200884 140024818968128 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-21:29:53.721287 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228813.sst) /dev/shm/ToplingTemp/Topling-297886-98UDNV 2024/11/22-21:29:54.264544 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228813.sst): first pass time = 0.54's, 247.080'MB/sec 2024/11/22-21:29:55.591911 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228813.sst): old prealloc_size = 147639500, real_size = 30933360 2024/11/22-21:29:55.592457 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228813.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924375 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0289 GB avg-val = 117.20 avg-zval = 31.31 2024/11/22-21:29:55.593256 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228813.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:29:56.437089 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228813.sst): second pass time = 1.33's, 81.662'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 41.442'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3333.087'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.322'MB/sec write SST data time = 0.13's, 277.934'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 24569.170'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.662'MB/sec zip pipeline throughput = 81.662'MB/sec entries = 924375 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.20 avg-zval = 31.31 usrkeys = 924375 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.20 avg-zval = 31.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924375 bytes = 3235328 } seq expand size = 6470625 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0289 GB dict = 1.08 MB all = 0.0350 GB } UnZip/Zip{ index = 10.5119 value = 3.7430 dict = 1.00 all = 3.8320 } Zip/UnZip{ index = 0.0951 value = 0.2672 dict = 1.00 all = 0.2610 } ---------------------------- total value len = 0.114806 GB avg = 0.124 KB (by entry num) total key len = 0.025882 GB avg = 0.028 KB total ukey len = 0.018488 GB avg = 0.020 KB total ukey num = 0.000924375 Billion total entry num = 0.000924375 Billion write speed all = 0.003095226 MB/sec (with seq num) write speed all = 0.002932532 MB/sec (without seq num) 2024/11/22-21:29:56.473764 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228813: 924375 keys, 35025260 bytes, temperature: kWarm 2024/11/22-21:29:56.473857 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282196473804, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228813, "file_size": 35025260, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774735, "largest_seqno": 3575802211, "table_properties": {"data_size": 28943392, "index_size": 1758720, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25882500, "raw_average_key_size": 28, "raw_value_size": 108335254, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924375, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282193, "oldest_key_time": 0, "file_creation_time": 1732282193, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228813, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:29:56.508843 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228814.sst) /dev/shm/ToplingTemp/Topling-297886-TKYHeX 2024/11/22-21:29:57.394419 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228814.sst): first pass time = 0.89's, 151.567'MB/sec 2024/11/22-21:29:58.701772 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228814.sst): old prealloc_size = 147639500, real_size = 31324992 2024/11/22-21:29:58.702324 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228814.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1085 GB zip-val = 0.0293 GB avg-val = 118.00 avg-zval = 31.90 2024/11/22-21:29:58.703106 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228814.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:29:59.561217 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228814.sst): second pass time = 1.31's, 83.014'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 41.192'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3408.801'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.314'MB/sec write SST data time = 0.13's, 281.271'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 31694.856'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 83.014'MB/sec zip pipeline throughput = 83.014'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 31.90 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 31.90 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3217568 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0293 GB dict = 1.09 MB all = 0.0354 GB } UnZip/Zip{ index = 10.3802 value = 3.6993 dict = 1.00 all = 3.7913 } Zip/UnZip{ index = 0.0963 value = 0.2703 dict = 1.00 all = 0.2638 } ---------------------------- total value len = 0.229718 GB avg = 0.125 KB (by entry num) total key len = 0.051623 GB avg = 0.028 KB total ukey len = 0.036873 GB avg = 0.020 KB total ukey num = 0.001843675 Billion total entry num = 0.001843675 Billion write speed all = 0.006189247 MB/sec (with seq num) write speed all = 0.005864774 MB/sec (without seq num) 2024/11/22-21:29:59.598798 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228814: 919300 keys, 35401714 bytes, temperature: kWarm 2024/11/22-21:29:59.598861 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282199598829, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228814, "file_size": 35401714, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3377703869, "largest_seqno": 3564306111, "table_properties": {"data_size": 29323696, "index_size": 1771264, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282196, "oldest_key_time": 0, "file_creation_time": 1732282196, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228814, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:29:59.635755 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228815.sst) /dev/shm/ToplingTemp/Topling-297886-CLB9tV 2024/11/22-21:30:00.573549 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228815.sst): first pass time = 0.94's, 143.135'MB/sec 2024/11/22-21:30:01.914737 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228815.sst): old prealloc_size = 147639500, real_size = 35170848 2024/11/22-21:30:01.915283 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228815.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928581 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0332 GB avg-val = 116.54 avg-zval = 35.71 2024/11/22-21:30:01.916043 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228815.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:30:02.798347 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228815.sst): second pass time = 1.34's, 80.728'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 44.448'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3390.968'MB/sec (index lex order gen) rebuild zvType time = 0.76's, 0.306'MB/sec write SST data time = 0.13's, 313.117'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 36692.004'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.728'MB/sec zip pipeline throughput = 80.728'MB/sec entries = 928581 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.54 avg-zval = 35.71 usrkeys = 928581 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.54 avg-zval = 35.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928581 bytes = 3250048 } seq expand size = 6500067 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.4472 value = 3.2634 dict = 1.00 all = 3.4175 } Zip/UnZip{ index = 0.0957 value = 0.3064 dict = 1.00 all = 0.2926 } ---------------------------- total value len = 0.344436 GB avg = 0.124 KB (by entry num) total key len = 0.077623 GB avg = 0.028 KB total ukey len = 0.055445 GB avg = 0.020 KB total ukey num = 0.002772256 Billion total entry num = 0.002772256 Billion write speed all = 0.009284248 MB/sec (with seq num) write speed all = 0.008796386 MB/sec (without seq num) 2024/11/22-21:30:02.839069 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228815: 928581 keys, 39273908 bytes, temperature: kWarm 2024/11/22-21:30:02.839132 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282202839101, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228815, "file_size": 39273908, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3575802110, "table_properties": {"data_size": 33160848, "index_size": 1777664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26000268, "raw_average_key_size": 28, "raw_value_size": 108217522, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928581, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282199, "oldest_key_time": 0, "file_creation_time": 1732282199, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228815, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:02.876085 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228816.sst) /dev/shm/ToplingTemp/Topling-297886-FtzoDU 2024/11/22-21:30:03.716086 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228816.sst): first pass time = 0.84's, 159.791'MB/sec 2024/11/22-21:30:05.051616 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228816.sst): old prealloc_size = 147639500, real_size = 32656880 2024/11/22-21:30:05.052169 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228816.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923869 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0307 GB avg-val = 117.28 avg-zval = 33.20 2024/11/22-21:30:05.052922 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228816.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:30:05.960918 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228816.sst): second pass time = 1.33's, 81.199'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 40.414'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3334.588'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.295'MB/sec write SST data time = 0.12's, 294.904'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 23534.071'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.199'MB/sec zip pipeline throughput = 81.199'MB/sec entries = 923869 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.28 avg-zval = 33.20 usrkeys = 923869 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.28 avg-zval = 33.20 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923869 bytes = 3233552 } seq expand size = 6467083 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5150 value = 3.5329 dict = 1.00 all = 3.6524 } Zip/UnZip{ index = 0.0951 value = 0.2831 dict = 1.00 all = 0.2738 } ---------------------------- total value len = 0.459252 GB avg = 0.124 KB (by entry num) total key len = 0.103492 GB avg = 0.028 KB total ukey len = 0.073923 GB avg = 0.020 KB total ukey num = 0.003696125 Billion total entry num = 0.003696125 Billion write speed all = 0.012378101 MB/sec (with seq num) write speed all = 0.011727702 MB/sec (without seq num) 2024/11/22-21:30:06.000565 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228816: 923869 keys, 36747386 bytes, temperature: kWarm 2024/11/22-21:30:06.000628 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282206000596, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228816, "file_size": 36747386, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774802, "largest_seqno": 3575802133, "table_properties": {"data_size": 30668512, "index_size": 1757248, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25868332, "raw_average_key_size": 28, "raw_value_size": 108349398, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923869, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282202, "oldest_key_time": 0, "file_creation_time": 1732282202, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228816, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:06.035914 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228817.sst) /dev/shm/ToplingTemp/Topling-297886-8LLY5W 2024/11/22-21:30:06.918037 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228817.sst): first pass time = 0.88's, 152.162'MB/sec 2024/11/22-21:30:08.308079 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228817.sst): old prealloc_size = 147639500, real_size = 34925328 2024/11/22-21:30:08.308623 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228817.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 973350 prefix = 4 raw-key = 0.0156 GB zip-key = 0.0019 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1070 GB zip-val = 0.0328 GB avg-val = 109.89 avg-zval = 33.72 2024/11/22-21:30:08.309053 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228817.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0024 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:30:09.084562 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228817.sst): second pass time = 1.39's, 76.989'MB/sec, value only(79.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 47.960'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3576.077'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.377'MB/sec write SST data time = 0.13's, 283.891'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.069'MB, throughput = 28211.087'MB/sec zip my value time = 1.39's, unzip length = 0.107'GB zip my value throughput = 76.989'MB/sec zip pipeline throughput = 76.989'MB/sec entries = 973350 avg-key = 28.00 avg-zkey = 1.91 avg-val = 109.89 avg-zval = 33.72 usrkeys = 973350 avg-key = 20.00 avg-zkey = 1.91 avg-val = 112.83 avg-zval = 33.72 TagRS{ kinds = RS_Key0_TagN bytes = 152200 } TagArray{ size = 408104 bytes = 1326352 } seq expand size = 2856728 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0195 GB value = 0.1070 GB dict = 1.07 MB all = 0.1342 GB } __ZipSize{ index = 0.0019 GB value = 0.0328 GB dict = 1.07 MB all = 0.0372 GB } UnZip/Zip{ index = 10.4451 value = 3.2593 dict = 1.00 all = 3.6042 } Zip/UnZip{ index = 0.0957 value = 0.3068 dict = 1.00 all = 0.2775 } ---------------------------- total value len = 0.569073 GB avg = 0.122 KB (by entry num) total key len = 0.130745 GB avg = 0.028 KB total ukey len = 0.093390 GB avg = 0.020 KB total ukey num = 0.004669475 Billion total entry num = 0.004669475 Billion write speed all = 0.015392131 MB/sec (with seq num) write speed all = 0.014570510 MB/sec (without seq num) 2024/11/22-21:30:09.122980 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228817: 973350 keys, 37239683 bytes, temperature: kWarm 2024/11/22-21:30:09.123042 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282209123010, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228817, "file_size": 37239683, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3575801838, "table_properties": {"data_size": 32818048, "index_size": 1863744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 27253800, "raw_average_key_size": 28, "raw_value_size": 106963974, "raw_average_value_size": 109, "num_data_blocks": 1, "num_entries": 973350, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282206, "oldest_key_time": 0, "file_creation_time": 1732282206, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228817, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:09.156899 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228818.sst) /dev/shm/ToplingTemp/Topling-297886-K4KNsY 2024/11/22-21:30:10.031854 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228818.sst): first pass time = 0.87's, 153.408'MB/sec 2024/11/22-21:30:11.400850 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228818.sst): old prealloc_size = 147639500, real_size = 35797952 2024/11/22-21:30:11.401423 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228818.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1035143 prefix = 4 raw-key = 0.0166 GB zip-key = 0.0020 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1052 GB zip-val = 0.0336 GB avg-val = 101.66 avg-zval = 32.43 2024/11/22-21:30:11.401712 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228818.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0026 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:30:12.081920 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228818.sst): second pass time = 1.37's, 76.908'MB/sec, value only(78.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.68's, 55.046'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3642.185'MB/sec (index lex order gen) rebuild zvType time = 0.54's, 0.475'MB/sec write SST data time = 0.14's, 276.156'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.035'MB, throughput = 33029.783'MB/sec zip my value time = 1.37's, unzip length = 0.105'GB zip my value throughput = 76.908'MB/sec zip pipeline throughput = 76.908'MB/sec entries = 1035143 avg-key = 28.00 avg-zkey = 1.90 avg-val = 101.66 avg-zval = 32.43 usrkeys = 1035143 avg-key = 20.00 avg-zkey = 1.90 avg-val = 103.09 avg-zval = 32.43 TagRS{ kinds = RS_Key0_TagN bytes = 161840 } TagArray{ size = 211986 bytes = 741968 } seq expand size = 1483902 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0207 GB value = 0.1052 GB dict = 1.03 MB all = 0.1342 GB } __ZipSize{ index = 0.0020 GB value = 0.0336 GB dict = 1.03 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5150 value = 3.1347 dict = 1.00 all = 3.5802 } Zip/UnZip{ index = 0.0951 value = 0.3190 dict = 1.00 all = 0.2793 } ---------------------------- total value len = 0.675791 GB avg = 0.118 KB (by entry num) total key len = 0.159729 GB avg = 0.028 KB total ukey len = 0.114092 GB avg = 0.020 KB total ukey num = 0.005704618 Billion total entry num = 0.005704618 Billion write speed all = 0.018375604 MB/sec (with seq num) write speed all = 0.017371910 MB/sec (without seq num) 2024/11/22-21:30:12.122474 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228818: 1035143 keys, 37488563 bytes, temperature: kWarm 2024/11/22-21:30:12.122536 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282212122504, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228818, "file_size": 37488563, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3422206562, "table_properties": {"data_size": 33570096, "index_size": 1968896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 28984004, "raw_average_key_size": 28, "raw_value_size": 105233737, "raw_average_value_size": 101, "num_data_blocks": 1, "num_entries": 1035143, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282209, "oldest_key_time": 0, "file_creation_time": 1732282209, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228818, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:12.157502 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228819.sst) /dev/shm/ToplingTemp/Topling-297886-Ltt7LY 2024/11/22-21:30:13.085000 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228819.sst): first pass time = 0.93's, 144.715'MB/sec 2024/11/22-21:30:14.436224 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228819.sst): old prealloc_size = 147639500, real_size = 33514288 2024/11/22-21:30:14.436732 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228819.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921936 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0315 GB avg-val = 117.58 avg-zval = 34.19 2024/11/22-21:30:14.437528 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228819.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:30:15.350131 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228819.sst): second pass time = 1.35's, 80.262'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 41.142'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3606.089'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.293'MB/sec write SST data time = 0.13's, 297.223'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 33970.589'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.262'MB/sec zip pipeline throughput = 80.262'MB/sec entries = 921936 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.58 avg-zval = 34.19 usrkeys = 921936 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.58 avg-zval = 34.19 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921936 bytes = 3226784 } seq expand size = 6453552 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.4458 value = 3.4394 dict = 1.00 all = 3.5696 } Zip/UnZip{ index = 0.0957 value = 0.2908 dict = 1.00 all = 0.2801 } ---------------------------- total value len = 0.790648 GB avg = 0.119 KB (by entry num) total key len = 0.185544 GB avg = 0.028 KB total ukey len = 0.132531 GB avg = 0.020 KB total ukey num = 0.006626554 Billion total entry num = 0.006626554 Billion write speed all = 0.021467848 MB/sec (with seq num) write speed all = 0.020302028 MB/sec (without seq num) 2024/11/22-21:30:15.389735 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228819: 921936 keys, 37599792 bytes, temperature: kWarm 2024/11/22-21:30:15.389815 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282215389765, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228819, "file_size": 37599792, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3422206563, "largest_seqno": 3575802177, "table_properties": {"data_size": 31518432, "index_size": 1765184, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25814208, "raw_average_key_size": 28, "raw_value_size": 108403568, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921936, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282212, "oldest_key_time": 0, "file_creation_time": 1732282212, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228819, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:15.427469 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228820.sst) /dev/shm/ToplingTemp/Topling-297886-fMhuUW 2024/11/22-21:30:16.377244 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228820.sst): first pass time = 0.95's, 141.321'MB/sec 2024/11/22-21:30:17.729932 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228820.sst): old prealloc_size = 147639500, real_size = 34028944 2024/11/22-21:30:17.730468 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228820.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929570 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0320 GB avg-val = 116.39 avg-zval = 34.45 2024/11/22-21:30:17.731273 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228820.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:30:18.651702 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228820.sst): second pass time = 1.35's, 80.020'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.498'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3440.741'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.292'MB/sec write SST data time = 0.13's, 303.811'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 26552.806'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.020'MB/sec zip pipeline throughput = 80.020'MB/sec entries = 929570 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.39 avg-zval = 34.45 usrkeys = 929570 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.39 avg-zval = 34.45 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929570 bytes = 3369712 } seq expand size = 6506990 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0383 GB } UnZip/Zip{ index = 10.5128 value = 3.3780 dict = 1.00 all = 3.5088 } Zip/UnZip{ index = 0.0951 value = 0.2960 dict = 1.00 all = 0.2850 } ---------------------------- total value len = 0.905345 GB avg = 0.120 KB (by entry num) total key len = 0.211571 GB avg = 0.028 KB total ukey len = 0.151122 GB avg = 0.020 KB total ukey num = 0.007556124 Billion total entry num = 0.007556124 Billion write speed all = 0.024560805 MB/sec (with seq num) write speed all = 0.023231541 MB/sec (without seq num) 2024/11/22-21:30:18.691774 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228820: 929570 keys, 38251632 bytes, temperature: kWarm 2024/11/22-21:30:18.691852 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282218691819, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228820, "file_size": 38251632, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3575801898, "table_properties": {"data_size": 32027904, "index_size": 1768448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26027960, "raw_average_key_size": 28, "raw_value_size": 108189868, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929570, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282215, "oldest_key_time": 0, "file_creation_time": 1732282215, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228820, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:18.729760 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228821.sst) /dev/shm/ToplingTemp/Topling-297886-EChVtV 2024/11/22-21:30:19.625679 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228821.sst): first pass time = 0.90's, 149.819'MB/sec 2024/11/22-21:30:20.906641 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228821.sst): old prealloc_size = 147639500, real_size = 32745296 2024/11/22-21:30:20.907186 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228821.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924636 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0308 GB avg-val = 117.16 avg-zval = 33.26 2024/11/22-21:30:20.907966 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228821.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:30:21.808953 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228821.sst): second pass time = 1.28's, 84.619'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 40.955'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3353.648'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.298'MB/sec write SST data time = 0.13's, 290.181'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33584.024'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.619'MB/sec zip pipeline throughput = 84.619'MB/sec entries = 924636 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.16 avg-zval = 33.26 usrkeys = 924636 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.16 avg-zval = 33.26 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924636 bytes = 3351824 } seq expand size = 6472452 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0308 GB dict = 1.08 MB all = 0.0370 GB } UnZip/Zip{ index = 10.5145 value = 3.5223 dict = 1.00 all = 3.6321 } Zip/UnZip{ index = 0.0951 value = 0.2839 dict = 1.00 all = 0.2753 } ---------------------------- total value len = 1.020145 GB avg = 0.120 KB (by entry num) total key len = 0.237461 GB avg = 0.028 KB total ukey len = 0.169615 GB avg = 0.020 KB total ukey num = 0.008480760 Billion total entry num = 0.008480760 Billion write speed all = 0.027652640 MB/sec (with seq num) write speed all = 0.026160820 MB/sec (without seq num) 2024/11/22-21:30:21.846538 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228821: 924636 keys, 36953196 bytes, temperature: kWarm 2024/11/22-21:30:21.846601 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282221846570, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228821, "file_size": 36953196, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3294443844, "largest_seqno": 3575802491, "table_properties": {"data_size": 30755200, "index_size": 1758784, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25889808, "raw_average_key_size": 28, "raw_value_size": 108327988, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924636, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282218, "oldest_key_time": 0, "file_creation_time": 1732282218, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228821, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:21.883785 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228822.sst) /dev/shm/ToplingTemp/Topling-297886-vxN9IY 2024/11/22-21:30:22.727364 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228822.sst): first pass time = 0.84's, 159.244'MB/sec 2024/11/22-21:30:24.011765 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228822.sst): old prealloc_size = 147639500, real_size = 33307600 2024/11/22-21:30:24.012351 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228822.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925154 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 117.08 avg-zval = 33.85 2024/11/22-21:30:24.013149 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228822.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:30:24.956253 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228822.sst): second pass time = 1.28's, 84.376'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.600'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3182.058'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.284'MB/sec write SST data time = 0.13's, 290.694'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33385.181'MB/sec zip my value time = 1.28's, unzip length = 0.108'GB zip my value throughput = 84.376'MB/sec zip pipeline throughput = 84.376'MB/sec entries = 925154 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.08 avg-zval = 33.85 usrkeys = 925154 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.08 avg-zval = 33.85 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925154 bytes = 3238048 } seq expand size = 6476078 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5104 value = 3.4588 dict = 1.00 all = 3.5886 } Zip/UnZip{ index = 0.0951 value = 0.2891 dict = 1.00 all = 0.2787 } ---------------------------- total value len = 1.134935 GB avg = 0.121 KB (by entry num) total key len = 0.263366 GB avg = 0.028 KB total ukey len = 0.188118 GB avg = 0.020 KB total ukey num = 0.009405914 Billion total entry num = 0.009405914 Billion write speed all = 0.030744132 MB/sec (with seq num) write speed all = 0.029089685 MB/sec (without seq num) 2024/11/22-21:30:25.000009 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228822: 925154 keys, 37401056 bytes, temperature: kWarm 2024/11/22-21:30:25.000071 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282225000040, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228822, "file_size": 37401056, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3575802231, "table_properties": {"data_size": 31315648, "index_size": 1760448, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25904312, "raw_average_key_size": 28, "raw_value_size": 108313432, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925154, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282221, "oldest_key_time": 0, "file_creation_time": 1732282221, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228822, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:25.042165 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228823.sst) /dev/shm/ToplingTemp/Topling-297886-eNyzvV 2024/11/22-21:30:25.843496 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228823.sst): first pass time = 0.80's, 167.502'MB/sec 2024/11/22-21:30:27.155237 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228823.sst): old prealloc_size = 147639500, real_size = 33950864 2024/11/22-21:30:27.155833 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228823.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927676 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0320 GB avg-val = 116.68 avg-zval = 34.45 2024/11/22-21:30:27.156600 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228823.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:30:28.068226 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228823.sst): second pass time = 1.31's, 82.559'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 41.679'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3070.111'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.296'MB/sec write SST data time = 0.13's, 293.140'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31455.546'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.559'MB/sec zip pipeline throughput = 82.559'MB/sec entries = 927676 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.68 avg-zval = 34.45 usrkeys = 927676 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.68 avg-zval = 34.45 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927676 bytes = 3246880 } seq expand size = 6493732 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5139 value = 3.3874 dict = 1.00 all = 3.5273 } Zip/UnZip{ index = 0.0951 value = 0.2952 dict = 1.00 all = 0.2835 } ---------------------------- total value len = 1.249671 GB avg = 0.121 KB (by entry num) total key len = 0.289341 GB avg = 0.028 KB total ukey len = 0.206672 GB avg = 0.020 KB total ukey num = 0.010333590 Billion total entry num = 0.010333590 Billion write speed all = 0.033835612 MB/sec (with seq num) write speed all = 0.032018117 MB/sec (without seq num) 2024/11/22-21:30:28.107763 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228823: 927676 keys, 38051470 bytes, temperature: kWarm 2024/11/22-21:30:28.107834 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282228107802, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228823, "file_size": 38051470, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3326330223, "largest_seqno": 3575801966, "table_properties": {"data_size": 31954112, "index_size": 1764672, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25974928, "raw_average_key_size": 28, "raw_value_size": 108242920, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927676, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282225, "oldest_key_time": 0, "file_creation_time": 1732282225, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228823, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:28.146030 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228824.sst) /dev/shm/ToplingTemp/Topling-297886-Ec4IJX 2024/11/22-21:30:29.077869 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228824.sst): first pass time = 0.93's, 144.041'MB/sec 2024/11/22-21:30:30.382684 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228824.sst): old prealloc_size = 147639500, real_size = 34000368 2024/11/22-21:30:30.383227 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228824.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929331 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0320 GB avg-val = 116.42 avg-zval = 34.43 2024/11/22-21:30:30.383991 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228824.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-21:30:31.297996 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228824.sst): second pass time = 1.30's, 82.963'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.632'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3379.473'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.297'MB/sec write SST data time = 0.13's, 289.656'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 31475.668'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 82.963'MB/sec zip pipeline throughput = 82.963'MB/sec entries = 929331 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.42 avg-zval = 34.43 usrkeys = 929331 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.42 avg-zval = 34.43 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929331 bytes = 3252672 } seq expand size = 6505317 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0381 GB } UnZip/Zip{ index = 10.5158 value = 3.3811 dict = 1.00 all = 3.5223 } Zip/UnZip{ index = 0.0951 value = 0.2958 dict = 1.00 all = 0.2839 } ---------------------------- total value len = 1.364373 GB avg = 0.121 KB (by entry num) total key len = 0.315362 GB avg = 0.028 KB total ukey len = 0.225258 GB avg = 0.020 KB total ukey num = 0.011262921 Billion total entry num = 0.011262921 Billion write speed all = 0.036926827 MB/sec (with seq num) write speed all = 0.034946019 MB/sec (without seq num) 2024/11/22-21:30:31.335919 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228824: 929331 keys, 38105436 bytes, temperature: kWarm 2024/11/22-21:30:31.335980 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282231335949, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228824, "file_size": 38105436, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3575802297, "table_properties": {"data_size": 32000352, "index_size": 1767488, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26021268, "raw_average_key_size": 28, "raw_value_size": 108196470, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929331, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282228, "oldest_key_time": 0, "file_creation_time": 1732282228, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228824, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:31.371258 140024818968128 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228825.sst) /dev/shm/ToplingTemp/Topling-297886-MyQwPW 2024/11/22-21:30:31.674224 140024818968128 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228825.sst): first pass time = 0.30's, 263.876'MB/sec 2024/11/22-21:30:32.433448 140024818968128 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228825.sst): old prealloc_size = 147639500, real_size = 21544912 2024/11/22-21:30:32.433840 140024818968128 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228825.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 555830 prefix = 4 raw-key = 0.0089 GB zip-key = 0.0011 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0644 GB zip-val = 0.0203 GB avg-val = 115.81 avg-zval = 36.61 2024/11/22-21:30:32.434052 140024818968128 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228825.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0014 GB, waited 0.000 sec, Key+Value = 0.080 GB 2024/11/22-21:30:32.866493 140024818968128 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00/228825.sst): second pass time = 0.76's, 84.852'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.43's, 52.513'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2844.628'MB/sec (index lex order gen) rebuild zvType time = 0.36's, 0.391'MB/sec write SST data time = 0.08's, 295.980'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.644'MB, throughput = 20415.727'MB/sec zip my value time = 0.76's, unzip length = 0.064'GB zip my value throughput = 84.852'MB/sec zip pipeline throughput = 84.852'MB/sec entries = 555830 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.81 avg-zval = 36.61 usrkeys = 555830 avg-key = 20.00 avg-zkey = 1.90 avg-val = 118.22 avg-zval = 36.61 TagRS{ kinds = RS_Key0_TagN bytes = 86960 } TagArray{ size = 190809 bytes = 596288 } seq expand size = 1335663 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0111 GB value = 0.0644 GB dict = 0.64 MB all = 0.0799 GB } __ZipSize{ index = 0.0011 GB value = 0.0203 GB dict = 0.64 MB all = 0.0227 GB } UnZip/Zip{ index = 10.5042 value = 3.1636 dict = 1.00 all = 3.5152 } Zip/UnZip{ index = 0.0952 value = 0.3161 dict = 1.00 all = 0.2845 } ---------------------------- total value len = 1.430081 GB avg = 0.121 KB (by entry num) total key len = 0.330925 GB avg = 0.028 KB total ukey len = 0.236375 GB avg = 0.020 KB total ukey num = 0.011818751 Billion total entry num = 0.011818751 Billion write speed all = 0.038712125 MB/sec (with seq num) write speed all = 0.036633636 MB/sec (without seq num) 2024/11/22-21:30:32.892065 140024818968128 [compaction_job.cc:1948] [default] [JOB 11556] Generated table #228825: 555830 keys, 22739777 bytes, temperature: kWarm 2024/11/22-21:30:32.892125 140024818968128 EVENT_LOG_v1 {"time_micros": 1732282232892095, "cf_name": "default", "job": 11556, "event": "table_file_creation", "file_number": 228825, "file_size": 22739777, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3575800409, "table_properties": {"data_size": 20347456, "index_size": 1058304, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 15563240, "raw_average_key_size": 28, "raw_value_size": 64371852, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 555830, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732282231, "oldest_key_time": 0, "file_creation_time": 1732282231, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 228825, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-21:30:32.912374 140024818968128 [compaction_job.cc:703] job-11556: subcompact[0], size: 0.470279 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-11556/att-00 [228813,228814,228815,228816,228817,228818,228819,228820,228821,228822,228823,228824,228825] 2024/11/22-21:30:32.992425 140024818968128 (Original Log Time 2024/11/22-21:29:53.193390) EVENT_LOG_v1 {"time_micros": 1732282193193350, "job": 11556, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [228751], "files_L3": [228809, 228810, 228708, 228709, 228710, 228711, 228712, 228713, 228714, 228715, 228679, 228680, 228681, 228789], "score": 0.999617, "input_data_size": 492784099, "oldest_snapshot_seqno": 3576294940} 2024/11/22-21:30:32.992428 140024818968128 (Original Log Time 2024/11/22-21:30:32.991686) [compaction_job.cc:2022] [default] [JOB 11556] Compacted 1@2 + 14@3 files to L3 => 470278873 bytes 2024/11/22-21:30:32.992430 140024818968128 (Original Log Time 2024/11/22-21:30:32.992357) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 13 0 0 0] max score 0.02, MB/sec: 12.4 rd, 11.8 wr, level 3, files in(1, 14) out(13 +0 blob) MB in(26.5, 443.5 +0.0 blob) out(448.5 +0.0 blob), read-write-amplify(34.7) write-amplify(17.0) OK, records in: 11886650, records dropped: 67899 output_compression: Snappy 2024/11/22-21:30:32.992436 140024818968128 (Original Log Time 2024/11/22-21:30:32.992404) EVENT_LOG_v1 {"time_micros": 1732282232992381, "cf": "default", "job": 11556, "event": "compaction_finished", "compaction_time_micros": 39718993, "compaction_time_cpu_micros": 17802797, "output_level": 3, "num_output_files": 13, "total_output_size": 470278873, "num_input_records": 11886650, "num_output_records": 11818751, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 294258064, "file_range_sync_nanos": 0, "file_fsync_nanos": 13546, "file_prepare_write_nanos": 29157, "lsm_state": [0, 0, 0, 13, 0, 0, 0]} 2024/11/22-21:30:33.110226 140024818968128 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732282193,"executesMs":39937,"compactionJobId":11556,"attempt":0,"compactionInputRawBytes":1700190069,"compactionInputZipBytes":492784099,"compactionOutputRawBytes":0,"compactionOutputZipBytes":470278873,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-df547ed66b57c4249d72a2a122d545e8-3f2310eb0eac5c10-00"}}, response = 2024/11/22-21:30:33.110298 140024818968128 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-11556/att-00: olev 3, work 39.938 s, result 6.400 ms, install 0.767 ms, input{raw 1.583 GiB zip 469.956 MiB}