2024/11/22-23:35:34.454750 140024852538944 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00: fork to child time = 0.009015 sec 2024/11/22-23:35:34.455621 140024852538944 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00 2024/11/22-23:35:34.462397 140024852538944 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00 2024/11/22-23:35:34.463546 140024852538944 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/22-23:35:34.463823 140024852538944 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/22-23:35:34.512795 140024852538944 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-14156/att-00/MANIFEST-206347 2024/11/22-23:35:34.542796 140024852538944 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-14156/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 234185, last_sequence is 3646970025, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 233971 2024/11/22-23:35:34.542806 140024852538944 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/22-23:35:34.542912 140024852538944 [version_set.cc:5721] Creating manifest 234185 2024/11/22-23:35:34.574053 140024852538944 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/22-23:35:34.596304 140024852538944 [compaction_job.cc:2437] [default] [JOB 14156] Compacting 1@2 + 13@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/22-23:35:34.596321 140024852538944 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 10616 Base level 2, inputs: [234109(27MB)], [234182(36MB) 234183(6577KB) 234080(6575KB) 234037(35MB) 234038(34MB) 234039(34MB) 234040(35MB) 234041(34MB) 234042(34MB) 234052(37MB) 234053(35MB) 234054(35MB) 234164(35MB)] 2024/11/22-23:35:34.602120 140024852538944 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-23:35:34.602153 140024852538944 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-23:35:34.602174 140024852538944 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/22-23:35:34.602179 140024852538944 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-23:35:34.602190 140024852538944 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/22-23:35:35.128740 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234186.sst) /dev/shm/ToplingTemp/Topling-335187-JgLQBs 2024/11/22-23:35:35.671401 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234186.sst): first pass time = 0.54's, 247.356'MB/sec 2024/11/22-23:35:37.026183 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234186.sst): old prealloc_size = 147639500, real_size = 34472384 2024/11/22-23:35:37.026721 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234186.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 929307 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0325 GB avg-val = 116.43 avg-zval = 34.94 2024/11/22-23:35:37.027544 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234186.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:35:37.934057 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234186.sst): second pass time = 1.35's, 79.937'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.91's, 42.621'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3405.979'MB/sec (index lex order gen) rebuild zvType time = 0.78's, 0.298'MB/sec write SST data time = 0.13's, 303.160'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.079'MB, throughput = 33453.351'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 79.937'MB/sec zip pipeline throughput = 79.937'MB/sec entries = 929307 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.43 avg-zval = 34.94 usrkeys = 929307 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.43 avg-zval = 34.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 929307 bytes = 3368752 } seq expand size = 6505149 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5159 value = 3.3320 dict = 1.00 all = 3.4687 } Zip/UnZip{ index = 0.0951 value = 0.3001 dict = 1.00 all = 0.2883 } ---------------------------- total value len = 0.114702 GB avg = 0.123 KB (by entry num) total key len = 0.026021 GB avg = 0.028 KB total ukey len = 0.018586 GB avg = 0.020 KB total ukey num = 0.000929307 Billion total entry num = 0.000929307 Billion write speed all = 0.002655408 MB/sec (with seq num) write speed all = 0.002515121 MB/sec (without seq num) 2024/11/22-23:35:37.975552 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234186: 929307 keys, 38693880 bytes, temperature: kWarm 2024/11/22-23:35:37.975641 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289737975587, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234186, "file_size": 38693880, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3376774746, "largest_seqno": 3646405244, "table_properties": {"data_size": 32472432, "index_size": 1767424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26020596, "raw_average_key_size": 28, "raw_value_size": 108197146, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 929307, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289735, "oldest_key_time": 0, "file_creation_time": 1732289735, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234186, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:35:38.010318 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234187.sst) /dev/shm/ToplingTemp/Topling-335187-YVPOLt 2024/11/22-23:35:38.903608 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234187.sst): first pass time = 0.89's, 150.258'MB/sec 2024/11/22-23:35:40.227056 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234187.sst): old prealloc_size = 147639500, real_size = 33345808 2024/11/22-23:35:40.227594 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234187.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926668 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.84 avg-zval = 33.83 2024/11/22-23:35:40.228356 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234187.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:35:41.170612 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234187.sst): second pass time = 1.32's, 81.885'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.94's, 39.562'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3393.859'MB/sec (index lex order gen) rebuild zvType time = 0.81's, 0.284'MB/sec write SST data time = 0.13's, 289.988'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31676.565'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 81.885'MB/sec zip pipeline throughput = 81.885'MB/sec entries = 926668 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.84 avg-zval = 33.83 usrkeys = 926668 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.84 avg-zval = 33.83 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926668 bytes = 3127520 } seq expand size = 6486676 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0373 GB } UnZip/Zip{ index = 10.5124 value = 3.4535 dict = 1.00 all = 3.5956 } Zip/UnZip{ index = 0.0951 value = 0.2896 dict = 1.00 all = 0.2781 } ---------------------------- total value len = 0.229460 GB avg = 0.124 KB (by entry num) total key len = 0.051967 GB avg = 0.028 KB total ukey len = 0.037119 GB avg = 0.020 KB total ukey num = 0.001855975 Billion total entry num = 0.001855975 Billion write speed all = 0.005310144 MB/sec (with seq num) write speed all = 0.005029987 MB/sec (without seq num) 2024/11/22-23:35:41.204600 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234187: 926668 keys, 37328150 bytes, temperature: kWarm 2024/11/22-23:35:41.204663 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289741204630, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234187, "file_size": 37328150, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3563652023, "largest_seqno": 3646404719, "table_properties": {"data_size": 31350976, "index_size": 1763008, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25946704, "raw_average_key_size": 28, "raw_value_size": 108271148, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926668, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289738, "oldest_key_time": 0, "file_creation_time": 1732289738, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234187, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:35:41.240739 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234188.sst) /dev/shm/ToplingTemp/Topling-335187-CDuxwr 2024/11/22-23:35:42.137840 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234188.sst): first pass time = 0.90's, 149.621'MB/sec 2024/11/22-23:35:43.493959 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234188.sst): old prealloc_size = 147639500, real_size = 33663648 2024/11/22-23:35:43.494497 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234188.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930953 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0317 GB avg-val = 116.17 avg-zval = 34.01 2024/11/22-23:35:43.495389 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234188.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:35:44.410025 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234188.sst): second pass time = 1.36's, 79.791'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.235'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3430.999'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.296'MB/sec write SST data time = 0.13's, 294.303'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33943.701'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.791'MB/sec zip pipeline throughput = 79.791'MB/sec entries = 930953 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.17 avg-zval = 34.01 usrkeys = 930953 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.17 avg-zval = 34.01 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930953 bytes = 3258352 } seq expand size = 6516671 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0378 GB } UnZip/Zip{ index = 10.5171 value = 3.4160 dict = 1.00 all = 3.5533 } Zip/UnZip{ index = 0.0951 value = 0.2927 dict = 1.00 all = 0.2814 } ---------------------------- total value len = 0.344128 GB avg = 0.123 KB (by entry num) total key len = 0.078034 GB avg = 0.028 KB total ukey len = 0.055739 GB avg = 0.020 KB total ukey num = 0.002786928 Billion total entry num = 0.002786928 Billion write speed all = 0.007965120 MB/sec (with seq num) write speed all = 0.007544462 MB/sec (without seq num) 2024/11/22-23:35:44.444813 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234188: 930953 keys, 37773074 bytes, temperature: kWarm 2024/11/22-23:35:44.444878 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289744444845, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234188, "file_size": 37773074, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3504428200, "largest_seqno": 3646404737, "table_properties": {"data_size": 31660368, "index_size": 1770368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26066684, "raw_average_key_size": 28, "raw_value_size": 108151066, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930953, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289741, "oldest_key_time": 0, "file_creation_time": 1732289741, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234188, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:35:44.485986 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234189.sst) /dev/shm/ToplingTemp/Topling-335187-GElA9u 2024/11/22-23:35:45.318883 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234189.sst): first pass time = 0.83's, 161.154'MB/sec 2024/11/22-23:35:46.641832 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234189.sst): old prealloc_size = 147639500, real_size = 34110448 2024/11/22-23:35:46.642441 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234189.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 1085202 prefix = 4 raw-key = 0.0174 GB zip-key = 0.0021 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1038 GB zip-val = 0.0318 GB avg-val = 95.68 avg-zval = 29.27 2024/11/22-23:35:46.642809 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234189.sst: sumWaitingMem = 0.003 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0028 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:35:47.378862 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234189.sst): second pass time = 1.32's, 78.534'MB/sec, value only(77.4% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.74's, 49.068'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3535.309'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.456'MB/sec write SST data time = 0.14's, 254.789'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.036'MB, throughput = 32346.317'MB/sec zip my value time = 1.32's, unzip length = 0.104'GB zip my value throughput = 78.534'MB/sec zip pipeline throughput = 78.534'MB/sec entries = 1085202 avg-key = 28.00 avg-zkey = 1.91 avg-val = 95.68 avg-zval = 29.27 usrkeys = 1085202 avg-key = 20.00 avg-zkey = 1.91 avg-val = 97.88 avg-zval = 29.27 TagRS{ kinds = RS_Key0_TagN bytes = 169680 } TagArray{ size = 340713 bytes = 1107328 } seq expand size = 2384991 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0217 GB value = 0.1038 GB dict = 1.04 MB all = 0.1342 GB } __ZipSize{ index = 0.0021 GB value = 0.0318 GB dict = 1.04 MB all = 0.0362 GB } UnZip/Zip{ index = 10.4478 value = 3.2691 dict = 1.00 all = 3.7114 } Zip/UnZip{ index = 0.0957 value = 0.3059 dict = 1.00 all = 0.2694 } ---------------------------- total value len = 0.450345 GB avg = 0.116 KB (by entry num) total key len = 0.108420 GB avg = 0.028 KB total ukey len = 0.077443 GB avg = 0.020 KB total ukey num = 0.003872130 Billion total entry num = 0.003872130 Billion write speed all = 0.010541878 MB/sec (with seq num) write speed all = 0.009957452 MB/sec (without seq num) 2024/11/22-23:35:47.420522 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234189: 1085202 keys, 36163495 bytes, temperature: kWarm 2024/11/22-23:35:47.420585 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289747420553, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234189, "file_size": 36163495, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3646404750, "table_properties": {"data_size": 31761568, "index_size": 2077376, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 30385656, "raw_average_key_size": 28, "raw_value_size": 103832181, "raw_average_value_size": 95, "num_data_blocks": 1, "num_entries": 1085202, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289744, "oldest_key_time": 0, "file_creation_time": 1732289744, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234189, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:35:47.455313 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234190.sst) /dev/shm/ToplingTemp/Topling-335187-fjmUPs 2024/11/22-23:35:48.216766 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234190.sst): first pass time = 0.76's, 176.276'MB/sec 2024/11/22-23:35:49.677870 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234190.sst): old prealloc_size = 147639500, real_size = 36554848 2024/11/22-23:35:49.678400 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234190.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924586 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0346 GB avg-val = 117.17 avg-zval = 37.38 2024/11/22-23:35:49.678725 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234190.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:35:50.396691 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234190.sst): second pass time = 1.46's, 74.179'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.72's, 53.619'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3436.854'MB/sec (index lex order gen) rebuild zvType time = 0.59's, 0.391'MB/sec write SST data time = 0.13's, 302.454'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 34297.022'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.179'MB/sec zip pipeline throughput = 74.179'MB/sec entries = 924586 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.17 avg-zval = 37.38 usrkeys = 924586 avg-key = 20.00 avg-zkey = 1.90 avg-val = 119.29 avg-zval = 37.38 TagRS{ kinds = RS_Key0_TagN bytes = 144560 } TagArray{ size = 280672 bytes = 982368 } seq expand size = 1964704 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0346 GB dict = 1.08 MB all = 0.0385 GB } UnZip/Zip{ index = 10.5135 value = 3.1341 dict = 1.00 all = 3.4824 } Zip/UnZip{ index = 0.0951 value = 0.3191 dict = 1.00 all = 0.2872 } ---------------------------- total value len = 0.560639 GB avg = 0.117 KB (by entry num) total key len = 0.134308 GB avg = 0.028 KB total ukey len = 0.095934 GB avg = 0.020 KB total ukey num = 0.004796716 Billion total entry num = 0.004796716 Billion write speed all = 0.013110406 MB/sec (with seq num) write speed all = 0.012386473 MB/sec (without seq num) 2024/11/22-23:35:50.431044 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234190: 924586 keys, 38541791 bytes, temperature: kWarm 2024/11/22-23:35:50.431113 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289750431076, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234190, "file_size": 38541791, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3435377046, "table_properties": {"data_size": 34564688, "index_size": 1758848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25888408, "raw_average_key_size": 28, "raw_value_size": 108329436, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924586, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289747, "oldest_key_time": 0, "file_creation_time": 1732289747, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234190, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:35:50.469906 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234191.sst) /dev/shm/ToplingTemp/Topling-335187-ppAH1r 2024/11/22-23:35:51.267437 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234191.sst): first pass time = 0.80's, 168.299'MB/sec 2024/11/22-23:35:52.564509 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234191.sst): old prealloc_size = 147639500, real_size = 31201696 2024/11/22-23:35:52.565122 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234191.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 921908 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0292 GB avg-val = 117.59 avg-zval = 31.67 2024/11/22-23:35:52.565865 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234191.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:35:53.418621 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234191.sst): second pass time = 1.30's, 83.621'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 41.313'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2992.621'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 276.052'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 34735.276'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.621'MB/sec zip pipeline throughput = 83.621'MB/sec entries = 921908 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.59 avg-zval = 31.67 usrkeys = 921908 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.59 avg-zval = 31.67 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 921908 bytes = 3226688 } seq expand size = 6453356 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0292 GB dict = 1.08 MB all = 0.0353 GB } UnZip/Zip{ index = 10.3777 value = 3.7132 dict = 1.00 all = 3.8038 } Zip/UnZip{ index = 0.0964 value = 0.2693 dict = 1.00 all = 0.2629 } ---------------------------- total value len = 0.675497 GB avg = 0.118 KB (by entry num) total key len = 0.160121 GB avg = 0.028 KB total ukey len = 0.114372 GB avg = 0.020 KB total ukey num = 0.005718624 Billion total entry num = 0.005718624 Billion write speed all = 0.015763315 MB/sec (with seq num) write speed all = 0.014900295 MB/sec (without seq num) 2024/11/22-23:35:53.449577 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234191: 921908 keys, 35285300 bytes, temperature: kWarm 2024/11/22-23:35:53.449639 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289753449608, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234191, "file_size": 35285300, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3435377047, "largest_seqno": 3646403875, "table_properties": {"data_size": 29194320, "index_size": 1776704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25813424, "raw_average_key_size": 28, "raw_value_size": 108404428, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 921908, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289750, "oldest_key_time": 0, "file_creation_time": 1732289750, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234191, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:35:53.484968 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234192.sst) /dev/shm/ToplingTemp/Topling-335187-ciVi6r 2024/11/22-23:35:54.397681 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234192.sst): first pass time = 0.91's, 147.061'MB/sec 2024/11/22-23:35:55.750871 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234192.sst): old prealloc_size = 147639500, real_size = 34425504 2024/11/22-23:35:55.751456 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234192.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926492 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0324 GB avg-val = 116.87 avg-zval = 34.99 2024/11/22-23:35:55.752253 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234192.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:35:56.669874 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234192.sst): second pass time = 1.35's, 80.091'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 42.045'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3189.130'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.294'MB/sec write SST data time = 0.13's, 298.855'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 36137.179'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.091'MB/sec zip pipeline throughput = 80.091'MB/sec entries = 926492 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.87 avg-zval = 34.99 usrkeys = 926492 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.87 avg-zval = 34.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926492 bytes = 3358544 } seq expand size = 6485444 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0324 GB dict = 1.08 MB all = 0.0386 GB } UnZip/Zip{ index = 10.4433 value = 3.3399 dict = 1.00 all = 3.4737 } Zip/UnZip{ index = 0.0958 value = 0.2994 dict = 1.00 all = 0.2879 } ---------------------------- total value len = 0.790258 GB avg = 0.119 KB (by entry num) total key len = 0.186063 GB avg = 0.028 KB total ukey len = 0.132902 GB avg = 0.020 KB total ukey num = 0.006645116 Billion total entry num = 0.006645116 Billion write speed all = 0.018416447 MB/sec (with seq num) write speed all = 0.017413667 MB/sec (without seq num) 2024/11/22-23:35:56.702769 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234192: 926492 keys, 38638484 bytes, temperature: kWarm 2024/11/22-23:35:56.702842 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289756702809, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234192, "file_size": 38638484, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3375334738, "largest_seqno": 3646405180, "table_properties": {"data_size": 32419344, "index_size": 1774336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25941776, "raw_average_key_size": 28, "raw_value_size": 108276028, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926492, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289753, "oldest_key_time": 0, "file_creation_time": 1732289753, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234192, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:35:56.745846 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234193.sst) /dev/shm/ToplingTemp/Topling-335187-9lBUmv 2024/11/22-23:35:57.556820 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234193.sst): first pass time = 0.81's, 165.510'MB/sec 2024/11/22-23:35:58.890666 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234193.sst): old prealloc_size = 147639500, real_size = 34564480 2024/11/22-23:35:58.891227 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234193.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931442 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0325 GB avg-val = 116.10 avg-zval = 34.94 2024/11/22-23:35:58.892004 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234193.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:35:59.792331 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234193.sst): second pass time = 1.33's, 81.110'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.90's, 43.023'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3306.842'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.302'MB/sec write SST data time = 0.13's, 300.832'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 33198.337'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.110'MB/sec zip pipeline throughput = 81.110'MB/sec entries = 931442 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.10 avg-zval = 34.94 usrkeys = 931442 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.10 avg-zval = 34.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931442 bytes = 3376496 } seq expand size = 6520094 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0325 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.4448 value = 3.3224 dict = 1.00 all = 3.4600 } Zip/UnZip{ index = 0.0957 value = 0.3010 dict = 1.00 all = 0.2890 } ---------------------------- total value len = 0.904916 GB avg = 0.119 KB (by entry num) total key len = 0.212144 GB avg = 0.028 KB total ukey len = 0.151531 GB avg = 0.020 KB total ukey num = 0.007576558 Billion total entry num = 0.007576558 Billion write speed all = 0.021069958 MB/sec (with seq num) write speed all = 0.019926686 MB/sec (without seq num) 2024/11/22-23:35:59.825315 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234193: 931442 keys, 38791720 bytes, temperature: kWarm 2024/11/22-23:35:59.825380 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289759825346, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234193, "file_size": 38791720, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3273701103, "largest_seqno": 3646405322, "table_properties": {"data_size": 32547888, "index_size": 1783552, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26080376, "raw_average_key_size": 28, "raw_value_size": 108137400, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931442, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289756, "oldest_key_time": 0, "file_creation_time": 1732289756, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234193, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:35:59.861971 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234194.sst) /dev/shm/ToplingTemp/Topling-335187-2YdFVs 2024/11/22-23:36:00.652232 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234194.sst): first pass time = 0.79's, 169.850'MB/sec 2024/11/22-23:36:01.997671 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234194.sst): old prealloc_size = 147639500, real_size = 33344032 2024/11/22-23:36:01.998236 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234194.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925251 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 117.06 avg-zval = 33.89 2024/11/22-23:36:02.001006 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234194.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:36:02.946107 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234194.sst): second pass time = 1.34's, 80.545'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.595'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3210.812'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.283'MB/sec write SST data time = 0.13's, 286.044'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 33676.245'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.545'MB/sec zip pipeline throughput = 80.545'MB/sec entries = 925251 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.06 avg-zval = 33.89 usrkeys = 925251 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.06 avg-zval = 33.89 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925251 bytes = 3354048 } seq expand size = 6476757 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.5142 value = 3.4546 dict = 1.00 all = 3.5741 } Zip/UnZip{ index = 0.0951 value = 0.2895 dict = 1.00 all = 0.2798 } ---------------------------- total value len = 1.019704 GB avg = 0.120 KB (by entry num) total key len = 0.238051 GB avg = 0.028 KB total ukey len = 0.170036 GB avg = 0.020 KB total ukey num = 0.008501809 Billion total entry num = 0.008501809 Billion write speed all = 0.023722326 MB/sec (with seq num) write speed all = 0.022439514 MB/sec (without seq num) 2024/11/22-23:36:02.982653 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234194: 925251 keys, 37553128 bytes, temperature: kWarm 2024/11/22-23:36:02.982719 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289762982685, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234194, "file_size": 37553128, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3326330223, "largest_seqno": 3646404867, "table_properties": {"data_size": 31352528, "index_size": 1760000, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25907028, "raw_average_key_size": 28, "raw_value_size": 108310826, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925251, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289759, "oldest_key_time": 0, "file_creation_time": 1732289759, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234194, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:36:03.021511 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234195.sst) /dev/shm/ToplingTemp/Topling-335187-8wkXWr 2024/11/22-23:36:03.925294 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234195.sst): first pass time = 0.90's, 148.514'MB/sec 2024/11/22-23:36:05.276470 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234195.sst): old prealloc_size = 147639500, real_size = 34027856 2024/11/22-23:36:05.277075 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234195.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 931307 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1081 GB zip-val = 0.0320 GB avg-val = 116.12 avg-zval = 34.37 2024/11/22-23:36:05.277814 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234195.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:36:06.192169 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234195.sst): second pass time = 1.35's, 80.077'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 41.523'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3081.361'MB/sec (index lex order gen) rebuild zvType time = 0.79's, 0.297'MB/sec write SST data time = 0.13's, 292.714'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 35726.017'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.077'MB/sec zip pipeline throughput = 80.077'MB/sec entries = 931307 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.12 avg-zval = 34.37 usrkeys = 931307 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.12 avg-zval = 34.37 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 931307 bytes = 3143184 } seq expand size = 6519149 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1081 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0320 GB dict = 1.08 MB all = 0.0380 GB } UnZip/Zip{ index = 10.4440 value = 3.3782 dict = 1.00 all = 3.5300 } Zip/UnZip{ index = 0.0957 value = 0.2960 dict = 1.00 all = 0.2833 } ---------------------------- total value len = 1.134364 GB avg = 0.120 KB (by entry num) total key len = 0.264127 GB avg = 0.028 KB total ukey len = 0.188662 GB avg = 0.020 KB total ukey num = 0.009433116 Billion total entry num = 0.009433116 Billion write speed all = 0.026375132 MB/sec (with seq num) write speed all = 0.024951886 MB/sec (without seq num) 2024/11/22-23:36:06.224485 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234195: 931307 keys, 38022120 bytes, temperature: kWarm 2024/11/22-23:36:06.224548 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289766224516, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234195, "file_size": 38022120, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3516470397, "largest_seqno": 3646404409, "table_properties": {"data_size": 32011392, "index_size": 1783424, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26076596, "raw_average_key_size": 28, "raw_value_size": 108141254, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 931307, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289763, "oldest_key_time": 0, "file_creation_time": 1732289763, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234195, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:36:06.264360 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234196.sst) /dev/shm/ToplingTemp/Topling-335187-e392lv 2024/11/22-23:36:07.150381 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234196.sst): first pass time = 0.89's, 151.491'MB/sec 2024/11/22-23:36:08.515853 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234196.sst): old prealloc_size = 147639500, real_size = 33243488 2024/11/22-23:36:08.516425 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234196.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926512 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0312 GB avg-val = 116.86 avg-zval = 33.73 2024/11/22-23:36:08.519121 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234196.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/22-23:36:09.467805 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234196.sst): second pass time = 1.36's, 79.357'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.95's, 39.348'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3212.595'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.282'MB/sec write SST data time = 0.13's, 285.338'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 32538.373'MB/sec zip my value time = 1.36's, unzip length = 0.108'GB zip my value throughput = 79.357'MB/sec zip pipeline throughput = 79.357'MB/sec entries = 926512 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 33.73 usrkeys = 926512 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 33.73 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926512 bytes = 3358624 } seq expand size = 6485584 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0312 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5155 value = 3.4649 dict = 1.00 all = 3.5833 } Zip/UnZip{ index = 0.0951 value = 0.2886 dict = 1.00 all = 0.2791 } ---------------------------- total value len = 1.249125 GB avg = 0.121 KB (by entry num) total key len = 0.290070 GB avg = 0.028 KB total ukey len = 0.207193 GB avg = 0.020 KB total ukey num = 0.010359628 Billion total entry num = 0.010359628 Billion write speed all = 0.029026963 MB/sec (with seq num) write speed all = 0.027464023 MB/sec (without seq num) 2024/11/22-23:36:09.505539 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234196: 926512 keys, 37456546 bytes, temperature: kWarm 2024/11/22-23:36:09.505604 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289769505572, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234196, "file_size": 37456546, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 3369353416, "largest_seqno": 3646405671, "table_properties": {"data_size": 31249488, "index_size": 1762176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942336, "raw_average_key_size": 28, "raw_value_size": 108275460, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926512, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289766, "oldest_key_time": 0, "file_creation_time": 1732289766, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234196, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:36:09.543522 140024852538944 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234197.sst) /dev/shm/ToplingTemp/Topling-335187-IoL0ru 2024/11/22-23:36:09.799024 140024852538944 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234197.sst): first pass time = 0.26's, 250.552'MB/sec 2024/11/22-23:36:10.385721 140024852538944 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234197.sst): old prealloc_size = 147639500, real_size = 17486480 2024/11/22-23:36:10.386048 140024852538944 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234197.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 445865 prefix = 4 raw-key = 0.0071 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.0515 GB zip-val = 0.0165 GB avg-val = 115.56 avg-zval = 37.04 2024/11/22-23:36:10.386322 140024852538944 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234197.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.064 GB 2024/11/22-23:36:10.753749 140024852538944 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00/234197.sst): second pass time = 0.59's, 87.905'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.37's, 51.419'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2798.224'MB/sec (index lex order gen) rebuild zvType time = 0.30's, 0.366'MB/sec write SST data time = 0.06's, 301.363'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.520'MB, throughput = 16975.925'MB/sec zip my value time = 0.59's, unzip length = 0.052'GB zip my value throughput = 87.905'MB/sec zip pipeline throughput = 87.905'MB/sec entries = 445865 avg-key = 28.00 avg-zkey = 1.93 avg-val = 115.56 avg-zval = 37.04 usrkeys = 445865 avg-key = 20.00 avg-zkey = 1.93 avg-val = 119.68 avg-zval = 37.04 TagRS{ kinds = RS_Key0_TagN bytes = 69760 } TagArray{ size = 262868 bytes = 952912 } seq expand size = 1840076 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0089 GB value = 0.0515 GB dict = 0.52 MB all = 0.0640 GB } __ZipSize{ index = 0.0009 GB value = 0.0165 GB dict = 0.52 MB all = 0.0189 GB } UnZip/Zip{ index = 10.3670 value = 3.1198 dict = 1.00 all = 3.3826 } Zip/UnZip{ index = 0.0965 value = 0.3205 dict = 1.00 all = 0.2956 } ---------------------------- total value len = 1.302488 GB avg = 0.121 KB (by entry num) total key len = 0.302554 GB avg = 0.028 KB total ukey len = 0.216110 GB avg = 0.020 KB total ukey num = 0.010805493 Billion total entry num = 0.010805493 Billion write speed all = 0.030268009 MB/sec (with seq num) write speed all = 0.028637842 MB/sec (without seq num) 2024/11/22-23:36:10.773039 140024852538944 [compaction_job.cc:1948] [default] [JOB 14156] Generated table #234197: 445865 keys, 18922707 bytes, temperature: kWarm 2024/11/22-23:36:10.773100 140024852538944 EVENT_LOG_v1 {"time_micros": 1732289770773069, "cf_name": "default", "job": 14156, "event": "table_file_creation", "file_number": 234197, "file_size": 18922707, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 3646405590, "table_properties": {"data_size": 16514688, "index_size": 860160, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 12484220, "raw_average_key_size": 28, "raw_value_size": 51522798, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 445865, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732289769, "oldest_key_time": 0, "file_creation_time": 1732289769, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 234197, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/22-23:36:10.790017 140024852538944 [compaction_job.cc:703] job-14156: subcompact[0], size: 0.433170 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-14156/att-00 [234186,234187,234188,234189,234190,234191,234192,234193,234194,234195,234196,234197] 2024/11/22-23:36:10.866599 140024852538944 (Original Log Time 2024/11/22-23:35:34.596382) EVENT_LOG_v1 {"time_micros": 1732289734596337, "job": 14156, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [234109], "files_L3": [234182, 234183, 234080, 234037, 234038, 234039, 234040, 234041, 234042, 234052, 234053, 234054, 234164], "score": 0.999824, "input_data_size": 451622143, "oldest_snapshot_seqno": 3646970000} 2024/11/22-23:36:10.866602 140024852538944 (Original Log Time 2024/11/22-23:36:10.865933) [compaction_job.cc:2022] [default] [JOB 14156] Compacted 1@2 + 13@3 files to L3 => 433170395 bytes 2024/11/22-23:36:10.866604 140024852538944 (Original Log Time 2024/11/22-23:36:10.866533) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 12.5 rd, 12.0 wr, level 3, files in(1, 13) out(12 +0 blob) MB in(27.1, 403.6 +0.0 blob) out(413.1 +0.0 blob), read-write-amplify(31.1) write-amplify(15.2) OK, records in: 10888066, records dropped: 82573 output_compression: Snappy 2024/11/22-23:36:10.866610 140024852538944 (Original Log Time 2024/11/22-23:36:10.866580) EVENT_LOG_v1 {"time_micros": 1732289770866557, "cf": "default", "job": 14156, "event": "compaction_finished", "compaction_time_micros": 36193643, "compaction_time_cpu_micros": 16415696, "output_level": 3, "num_output_files": 12, "total_output_size": 433170395, "num_input_records": 10888066, "num_output_records": 10805493, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 265891372, "file_range_sync_nanos": 0, "file_fsync_nanos": 10969, "file_prepare_write_nanos": 30047, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/22-23:36:10.980801 140024852538944 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732289734,"executesMs":36404,"compactionJobId":14156,"attempt":0,"compactionInputRawBytes":1552128341,"compactionInputZipBytes":451622143,"compactionOutputRawBytes":0,"compactionOutputZipBytes":433170395,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-b796d54ef8fa31a2c60362d651254b8f-b4b0a641fdacf5d9-00"}}, response = 2024/11/22-23:36:10.980888 140024852538944 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-14156/att-00: olev 3, work 36.405 s, result 6.413 ms, install 0.696 ms, input{raw 1.446 GiB zip 430.700 MiB}