2024/11/22-17:42:45.412608 140025112712768 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00: fork to child time = 0.009473 sec 2024/11/22-17:42:45.413411 140025112712768 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00 2024/11/22-17:42:45.421250 140025112712768 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00 2024/11/22-17:42:45.422416 140025112712768 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-17:42:45.422727 140025112712768 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-17:42:45.467861 140025112712768 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-06403/att-00/MANIFEST-206347 2024/11/22-17:42:45.481016 140025112712768 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-06403/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 218313, last_sequence is 3433251399, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 218112 2024/11/22-17:42:45.481023 140025112712768 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-17:42:45.481099 140025112712768 [version_set.cc:5721] Creating manifest 218313 2024/11/22-17:42:45.512820 140025112712768 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-17:42:45.528717 140025112712768 [compaction_job.cc:2437] [default] [JOB 6403] Compacting 1@2 + 7@3 files to L3, score 0.99, subcompactions 1 : 1 2024/11/22-17:42:45.528732 140025112712768 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 4914 Base level 2, inputs: [218237(16MB)], [218311(5644KB) 218204(36MB) 218205(36MB) 218206(34MB) 218207(37MB) 218208(36MB) 218287(36MB)] 2024/11/22-17:42:45.533622 140025112712768 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-17:42:45.533653 140025112712768 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-17:42:45.533671 140025112712768 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-17:42:45.533676 140025112712768 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-17:42:45.533686 140025112712768 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-17:42:45.713634 140025112712768 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218314.sst) /dev/shm/ToplingTemp/Topling-224799-J5egnr 2024/11/22-17:42:46.521656 140025112712768 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218314.sst): first pass time = 0.81's, 166.116'MB/sec 2024/11/22-17:42:47.863191 140025112712768 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218314.sst): old prealloc_size = 147639500, real_size = 34876256 2024/11/22-17:42:47.863720 140025112712768 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218314.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925481 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0329 GB avg-val = 117.02 avg-zval = 35.53 2024/11/22-17:42:47.864483 140025112712768 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218314.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:42:48.763840 140025112712768 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218314.sst): second pass time = 1.34's, 80.779'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 43.013'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3464.842'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.300'MB/sec write SST data time = 0.13's, 300.850'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34481.942'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.779'MB/sec zip pipeline throughput = 80.779'MB/sec entries = 925481 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.02 avg-zval = 35.53 usrkeys = 925481 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.02 avg-zval = 35.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925481 bytes = 3007824 } seq expand size = 6478367 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0329 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5149 value = 3.2935 dict = 1.00 all = 3.4646 } Zip/UnZip{ index = 0.0951 value = 0.3036 dict = 1.00 all = 0.2886 } ---------------------------- total value len = 0.114783 GB avg = 0.124 KB (by entry num) total key len = 0.025913 GB avg = 0.028 KB total ukey len = 0.018510 GB avg = 0.020 KB total ukey num = 0.000925481 Billion total entry num = 0.000925481 Billion write speed all = 0.004420839 MB/sec (with seq num) write speed all = 0.004188202 MB/sec (without seq num) 2024/11/22-17:42:48.800273 140025112712768 [compaction_job.cc:1948] [default] [JOB 6403] Generated table #218314: 925481 keys, 38739314 bytes, temperature: kWarm 2024/11/22-17:42:48.800371 140025112712768 EVENT_LOG_v1 {"time_micros": 1732268568800313, "cf_name": "default", "job": 6403, "event": "table_file_creation", "file_number": 218314, "file_size": 38739314, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774735, "largest_seqno": 3432528749, "table_properties": {"data_size": 32884368, "index_size": 1760320, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25913468, "raw_average_key_size": 28, "raw_value_size": 108304334, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925481, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732268565, "oldest_key_time": 0, "file_creation_time": 1732268565, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218314, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:42:48.835974 140025112712768 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218315.sst) /dev/shm/ToplingTemp/Topling-224799-aAKaNq 2024/11/22-17:42:49.706211 140025112712768 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218315.sst): first pass time = 0.87's, 154.238'MB/sec 2024/11/22-17:42:51.021244 140025112712768 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218315.sst): old prealloc_size = 147639500, real_size = 34621056 2024/11/22-17:42:51.021828 140025112712768 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218315.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926431 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0326 GB avg-val = 116.88 avg-zval = 35.22 2024/11/22-17:42:51.022582 140025112712768 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218315.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:42:51.925758 140025112712768 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218315.sst): second pass time = 1.31's, 82.382'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 42.550'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3137.614'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.298'MB/sec write SST data time = 0.13's, 300.993'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31135.973'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.382'MB/sec zip pipeline throughput = 82.382'MB/sec entries = 926431 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.88 avg-zval = 35.22 usrkeys = 926431 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.88 avg-zval = 35.22 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926431 bytes = 3010912 } seq expand size = 6485017 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5150 value = 3.3186 dict = 1.00 all = 3.4874 } Zip/UnZip{ index = 0.0951 value = 0.3013 dict = 1.00 all = 0.2867 } ---------------------------- total value len = 0.229545 GB avg = 0.124 KB (by entry num) total key len = 0.051854 GB avg = 0.028 KB total ukey len = 0.037038 GB avg = 0.020 KB total ukey num = 0.001851912 Billion total entry num = 0.001851912 Billion write speed all = 0.008841009 MB/sec (with seq num) write speed all = 0.008375541 MB/sec (without seq num) 2024/11/22-17:42:51.959374 140025112712768 [compaction_job.cc:1948] [default] [JOB 6403] Generated table #218315: 926431 keys, 38486580 bytes, temperature: kWarm 2024/11/22-17:42:51.959442 140025112712768 EVENT_LOG_v1 {"time_micros": 1732268571959407, "cf_name": "default", "job": 6403, "event": "table_file_creation", "file_number": 218315, "file_size": 38486580, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774786, "largest_seqno": 3432527895, "table_properties": {"data_size": 32627184, "index_size": 1762112, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25940068, "raw_average_key_size": 28, "raw_value_size": 108277714, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926431, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732268568, "oldest_key_time": 0, "file_creation_time": 1732268568, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218315, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:42:51.997589 140025112712768 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218316.sst) /dev/shm/ToplingTemp/Topling-224799-ZPh4Ot 2024/11/22-17:42:52.831841 140025112712768 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218316.sst): first pass time = 0.83's, 160.893'MB/sec 2024/11/22-17:42:54.225821 140025112712768 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218316.sst): old prealloc_size = 147639500, real_size = 35153520 2024/11/22-17:42:54.226432 140025112712768 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218316.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1082604 prefix = 4 raw-key = 0.0173 GB zip-key = 0.0021 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1039 GB zip-val = 0.0328 GB avg-val = 95.98 avg-zval = 30.29 2024/11/22-17:42:54.226621 140025112712768 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218316.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0028 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:42:54.856679 140025112712768 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218316.sst): second pass time = 1.39's, 74.574'MB/sec, value only(77.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.63's, 57.483'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3612.994'MB/sec (index lex order gen) rebuild zvType time = 0.49's, 0.551'MB/sec write SST data time = 0.14's, 260.490'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.031'MB, throughput = 30592.687'MB/sec zip my value time = 1.39's, unzip length = 0.104'GB zip my value throughput = 74.574'MB/sec zip pipeline throughput = 74.574'MB/sec entries = 1082604 avg-key = 28.00 avg-zkey = 1.93 avg-val = 95.98 avg-zval = 30.29 usrkeys = 1082604 avg-key = 20.00 avg-zkey = 1.93 avg-val = 96.34 avg-zval = 30.29 TagRS{ kinds = RS_Key0_TagN bytes = 169240 } TagArray{ size = 56430 bytes = 169312 } seq expand size = 395010 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0217 GB value = 0.1039 GB dict = 1.03 MB all = 0.1342 GB } __ZipSize{ index = 0.0021 GB value = 0.0328 GB dict = 1.03 MB all = 0.0363 GB } UnZip/Zip{ index = 10.3796 value = 3.1682 dict = 1.00 all = 3.7012 } Zip/UnZip{ index = 0.0963 value = 0.3156 dict = 1.00 all = 0.2702 } ---------------------------- total value len = 0.333845 GB avg = 0.114 KB (by entry num) total key len = 0.082166 GB avg = 0.028 KB total ukey len = 0.058690 GB avg = 0.020 KB total ukey num = 0.002934516 Billion total entry num = 0.002934516 Billion write speed all = 0.013069081 MB/sec (with seq num) write speed all = 0.012331574 MB/sec (without seq num) 2024/11/22-17:42:54.889069 140025112712768 [compaction_job.cc:1948] [default] [JOB 6403] Generated table #218316: 1082604 keys, 36263125 bytes, temperature: kWarm 2024/11/22-17:42:54.889135 140025112712768 EVENT_LOG_v1 {"time_micros": 1732268574889101, "cf_name": "default", "job": 6403, "event": "table_file_creation", "file_number": 218316, "file_size": 36263125, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3432528785, "table_properties": {"data_size": 32796640, "index_size": 2086016, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 30312912, "raw_average_key_size": 28, "raw_value_size": 103904957, "raw_average_value_size": 95, "num_data_blocks": 1, "num_entries": 1082604, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732268571, "oldest_key_time": 0, "file_creation_time": 1732268571, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218316, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:42:54.925302 140025112712768 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218317.sst) /dev/shm/ToplingTemp/Topling-224799-YgARir 2024/11/22-17:42:55.769739 140025112712768 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218317.sst): first pass time = 0.84's, 158.952'MB/sec 2024/11/22-17:42:57.141108 140025112712768 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218317.sst): old prealloc_size = 147639500, real_size = 35987952 2024/11/22-17:42:57.141625 140025112712768 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218317.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925135 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0340 GB avg-val = 117.08 avg-zval = 36.74 2024/11/22-17:42:57.142188 140025112712768 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218317.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:42:57.942704 140025112712768 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218317.sst): second pass time = 1.37's, 79.023'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.80's, 48.606'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3574.301'MB/sec (index lex order gen) rebuild zvType time = 0.67's, 0.344'MB/sec write SST data time = 0.13's, 301.591'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 26981.201'MB/sec zip my value time = 1.37's, unzip length = 0.108'GB zip my value throughput = 79.023'MB/sec zip pipeline throughput = 79.023'MB/sec entries = 925135 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.08 avg-zval = 36.74 usrkeys = 925135 avg-key = 20.00 avg-zkey = 1.91 avg-val = 121.34 avg-zval = 36.74 TagRS{ kinds = RS_Key0_TagN bytes = 144640 } TagArray{ size = 562906 bytes = 1970192 } seq expand size = 3940342 multi value expand size = 0 cnt WriteAppend = 16 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0390 GB } UnZip/Zip{ index = 10.4445 value = 3.1871 dict = 1.00 all = 3.4449 } Zip/UnZip{ index = 0.0957 value = 0.3138 dict = 1.00 all = 0.2903 } ---------------------------- total value len = 0.446100 GB avg = 0.116 KB (by entry num) total key len = 0.108070 GB avg = 0.028 KB total ukey len = 0.077193 GB avg = 0.020 KB total ukey num = 0.003859651 Billion total entry num = 0.003859651 Billion write speed all = 0.017407653 MB/sec (with seq num) write speed all = 0.016437735 MB/sec (without seq num) 2024/11/22-17:42:57.978439 140025112712768 [compaction_job.cc:1948] [default] [JOB 6403] Generated table #218317: 925135 keys, 38961772 bytes, temperature: kWarm 2024/11/22-17:42:57.978505 140025112712768 EVENT_LOG_v1 {"time_micros": 1732268577978471, "cf_name": "default", "job": 6403, "event": "table_file_creation", "file_number": 218317, "file_size": 38961772, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3432527539, "table_properties": {"data_size": 33984992, "index_size": 1771520, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25903780, "raw_average_key_size": 28, "raw_value_size": 108314010, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925135, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732268574, "oldest_key_time": 0, "file_creation_time": 1732268574, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218317, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:42:58.018414 140025112712768 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218318.sst) /dev/shm/ToplingTemp/Topling-224799-4Lyu6p 2024/11/22-17:42:58.861902 140025112712768 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218318.sst): first pass time = 0.84's, 159.131'MB/sec 2024/11/22-17:43:00.224506 140025112712768 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218318.sst): old prealloc_size = 147639500, real_size = 34581984 2024/11/22-17:43:00.225068 140025112712768 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218318.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924900 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0326 GB avg-val = 117.12 avg-zval = 35.24 2024/11/22-17:43:00.225865 140025112712768 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218318.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:43:01.135354 140025112712768 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218318.sst): second pass time = 1.36's, 79.537'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 42.461'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3273.648'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.295'MB/sec write SST data time = 0.13's, 302.803'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35515.787'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.537'MB/sec zip pipeline throughput = 79.537'MB/sec entries = 924900 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.12 avg-zval = 35.24 usrkeys = 924900 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.12 avg-zval = 35.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924900 bytes = 3237168 } seq expand size = 6474300 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5106 value = 3.3237 dict = 1.00 all = 3.4704 } Zip/UnZip{ index = 0.0951 value = 0.3009 dict = 1.00 all = 0.2881 } ---------------------------- total value len = 0.560895 GB avg = 0.117 KB (by entry num) total key len = 0.133967 GB avg = 0.028 KB total ukey len = 0.095691 GB avg = 0.020 KB total ukey num = 0.004784551 Billion total entry num = 0.004784551 Billion write speed all = 0.021824900 MB/sec (with seq num) write speed all = 0.020622677 MB/sec (without seq num) 2024/11/22-17:43:01.167566 140025112712768 [compaction_job.cc:1948] [default] [JOB 6403] Generated table #218318: 924900 keys, 38674674 bytes, temperature: kWarm 2024/11/22-17:43:01.167632 140025112712768 EVENT_LOG_v1 {"time_micros": 1732268581167599, "cf_name": "default", "job": 6403, "event": "table_file_creation", "file_number": 218318, "file_size": 38674674, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3432528543, "table_properties": {"data_size": 32590608, "index_size": 1759936, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25897200, "raw_average_key_size": 28, "raw_value_size": 108320540, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924900, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732268578, "oldest_key_time": 0, "file_creation_time": 1732268578, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218318, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:43:01.201833 140025112712768 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218319.sst) /dev/shm/ToplingTemp/Topling-224799-vTsFVr 2024/11/22-17:43:02.140555 140025112712768 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218319.sst): first pass time = 0.94's, 142.986'MB/sec 2024/11/22-17:43:03.445848 140025112712768 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218319.sst): old prealloc_size = 147639500, real_size = 33771520 2024/11/22-17:43:03.446379 140025112712768 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218319.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922204 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1084 GB zip-val = 0.0318 GB avg-val = 117.54 avg-zval = 34.45 2024/11/22-17:43:03.447173 140025112712768 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218319.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-17:43:04.373479 140025112712768 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218319.sst): second pass time = 1.30's, 83.090'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 40.812'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3479.434'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 294.277'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 30933.280'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.090'MB/sec zip pipeline throughput = 83.090'MB/sec entries = 922204 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.54 avg-zval = 34.45 usrkeys = 922204 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.54 avg-zval = 34.45 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922204 bytes = 3227728 } seq expand size = 6455428 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0318 GB dict = 1.08 MB all = 0.0379 GB } UnZip/Zip{ index = 10.4424 value = 3.4114 dict = 1.00 all = 3.5453 } Zip/UnZip{ index = 0.0958 value = 0.2931 dict = 1.00 all = 0.2821 } ---------------------------- total value len = 0.675746 GB avg = 0.118 KB (by entry num) total key len = 0.159789 GB avg = 0.028 KB total ukey len = 0.114135 GB avg = 0.020 KB total ukey num = 0.005706755 Billion total entry num = 0.005706755 Billion write speed all = 0.026240632 MB/sec (with seq num) write speed all = 0.024806831 MB/sec (without seq num) 2024/11/22-17:43:04.408352 140025112712768 [compaction_job.cc:1948] [default] [JOB 6403] Generated table #218319: 922204 keys, 37858108 bytes, temperature: kWarm 2024/11/22-17:43:04.408419 140025112712768 EVENT_LOG_v1 {"time_micros": 1732268584408385, "cf_name": "default", "job": 6403, "event": "table_file_creation", "file_number": 218319, "file_size": 37858108, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701122, "largest_seqno": 3432528302, "table_properties": {"data_size": 31774512, "index_size": 1766272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25821712, "raw_average_key_size": 28, "raw_value_size": 108396072, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922204, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732268581, "oldest_key_time": 0, "file_creation_time": 1732268581, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218319, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:43:04.442656 140025112712768 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218320.sst) /dev/shm/ToplingTemp/Topling-224799-iZXyzp 2024/11/22-17:43:04.560790 140025112712768 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218320.sst): first pass time = 0.12's, 236.208'MB/sec 2024/11/22-17:43:04.820805 140025112712768 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218320.sst): old prealloc_size = 147639500, real_size = 7990976 2024/11/22-17:43:04.820966 140025112712768 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218320.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 193880 prefix = 4 raw-key = 0.0031 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0225 GB zip-val = 0.0076 GB avg-val = 115.86 avg-zval = 39.04 2024/11/22-17:43:04.821154 140025112712768 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218320.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0004 GB, waited 0.000 sec, Key+Value = 0.028 GB 2024/11/22-17:43:04.997720 140025112712768 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00/218320.sst): second pass time = 0.26's, 86.589'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.18's, 49.800'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2644.455'MB/sec (index lex order gen) rebuild zvType time = 0.15's, 0.325'MB/sec write SST data time = 0.03's, 319.845'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.235'MB, throughput = 7808.862'MB/sec zip my value time = 0.26's, unzip length = 0.022'GB zip my value throughput = 86.589'MB/sec zip pipeline throughput = 86.589'MB/sec entries = 193880 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.86 avg-zval = 39.04 usrkeys = 193880 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.86 avg-zval = 39.04 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 193880 bytes = 630128 } seq expand size = 1357160 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0039 GB value = 0.0225 GB dict = 0.23 MB all = 0.0279 GB } __ZipSize{ index = 0.0004 GB value = 0.0076 GB dict = 0.23 MB all = 0.0088 GB } UnZip/Zip{ index = 10.4031 value = 2.9675 dict = 1.00 all = 3.1660 } Zip/UnZip{ index = 0.0961 value = 0.3370 dict = 1.00 all = 0.3159 } ---------------------------- total value len = 0.699566 GB avg = 0.119 KB (by entry num) total key len = 0.165218 GB avg = 0.028 KB total ukey len = 0.118013 GB avg = 0.020 KB total ukey num = 0.005900635 Billion total entry num = 0.005900635 Billion write speed all = 0.027158663 MB/sec (with seq num) write speed all = 0.025676180 MB/sec (without seq num) 2024/11/22-17:43:05.015666 140025112712768 [compaction_job.cc:1948] [default] [JOB 6403] Generated table #218320: 193880 keys, 8809549 bytes, temperature: kWarm 2024/11/22-17:43:05.015740 140025112712768 EVENT_LOG_v1 {"time_micros": 1732268585015706, "cf_name": "default", "job": 6403, "event": "table_file_creation", "file_number": 218320, "file_size": 8809549, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369194706, "largest_seqno": 3432528239, "table_properties": {"data_size": 7569584, "index_size": 372736, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 5428640, "raw_average_key_size": 28, "raw_value_size": 22462440, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 193880, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732268584, "oldest_key_time": 0, "file_creation_time": 1732268584, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 218320, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-17:43:05.028604 140025112712768 [compaction_job.cc:703] job-06403: subcompact[0], size: 0.237793 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-06403/att-00 [218314,218315,218316,218317,218318,218319,218320] 2024/11/22-17:43:05.073248 140025112712768 (Original Log Time 2024/11/22-17:42:45.528790) EVENT_LOG_v1 {"time_micros": 1732268565528745, "job": 6403, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [218237], "files_L3": [218311, 218204, 218205, 218206, 218207, 218208, 218287], "score": 0.991114, "input_data_size": 250909427, "oldest_snapshot_seqno": 3433251381} 2024/11/22-17:43:05.073252 140025112712768 (Original Log Time 2024/11/22-17:43:05.072729) [compaction_job.cc:2022] [default] [JOB 6403] Compacted 1@2 + 7@3 files to L3 => 237793122 bytes 2024/11/22-17:43:05.073254 140025112712768 (Original Log Time 2024/11/22-17:43:05.073182) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 7 0 0 0] max score 0.01, MB/sec: 12.9 rd, 12.2 wr, level 3, files in(1, 7) out(7 +0 blob) MB in(16.3, 223.0 +0.0 blob) out(226.8 +0.0 blob), read-write-amplify(28.6) write-amplify(13.9) OK, records in: 5969633, records dropped: 68998 output_compression: Snappy 2024/11/22-17:43:05.073260 140025112712768 (Original Log Time 2024/11/22-17:43:05.073227) EVENT_LOG_v1 {"time_micros": 1732268585073208, "cf": "default", "job": 6403, "event": "compaction_finished", "compaction_time_micros": 19499822, "compaction_time_cpu_micros": 8740380, "output_level": 3, "num_output_files": 7, "total_output_size": 237793122, "num_input_records": 5969633, "num_output_records": 5900635, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 147167059, "file_range_sync_nanos": 0, "file_fsync_nanos": 7008, "file_prepare_write_nanos": 18840, "lsm_state": [0, 0, 0, 7, 0, 0, 0]} 2024/11/22-17:43:05.200861 140025112712768 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732268565,"executesMs":19655,"compactionJobId":6403,"attempt":0,"compactionInputRawBytes":842995563,"compactionInputZipBytes":250909427,"compactionOutputRawBytes":0,"compactionOutputZipBytes":237793122,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-5eb6b88b49d96b113dccb34f5c879e29-b4a6e246c2bc8322-00"}}, response = 2024/11/22-17:43:05.200958 140025112712768 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-06403/att-00: olev 3, work 19.655 s, result 4.886 ms, install 0.550 ms, input{raw 803.943 MiB zip 239.286 MiB}