2024/08/21-03:02:47.426697 140539937289792 DEBG dcompact_worker.cpp:2211: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00: fork to child time = 0.057363 sec 2024/08/21-03:02:47.427194 140539937289792 DEBG dcompact_worker.cpp:1011: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00 2024/08/21-03:02:47.430870 140539937289792 DEBG dcompact_worker.cpp:1031: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00 2024/08/21-03:02:47.431162 140539937289792 TRAC dcompact_worker.cpp:1081: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/08/21-03:02:47.431381 140539937289792 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/08/21-03:02:47.479254 140539937289792 [version_set.cc:6139] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-08-20T02.42.00/.rocksdb/job-62716/att-00/MANIFEST-14629260 2024/08/21-03:02:47.493862 140539937289792 [version_set.cc:6182] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-08-20T02.42.00/.rocksdb/job-62716/att-00/MANIFEST-14629260 succeeded,manifest_file_number is 14629260, next_file_number is 14701440, last_sequence is 191370425237, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 14701310 2024/08/21-03:02:47.493868 140539937289792 [version_set.cc:6197] Column family [default] (ID 0), log number is 0 2024/08/21-03:02:47.493925 140539937289792 [version_set.cc:5652] Creating manifest 14701440 2024/08/21-03:02:47.527490 140539937289792 DEBG dcompact_worker.cpp:1215: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00: bottommost_level: fake = 1, rpc = 0 2024/08/21-03:02:47.548528 140539937289792 [compaction_job.cc:2433] [default] [JOB 62716] Compacting 1@1 + 7@2 files to L2, score 1.00, subcompactions 1 : 1 2024/08/21-03:02:47.548541 140539937289792 [compaction_job.cc:2441] [default]: Compaction start summary: Base version 31571 Base level 1, inputs: [14701420(31MB)], [14701382(20MB) 14701383(20MB) 14701384(20MB) 14701385(20MB) 14701386(20MB) 14701397(19MB) 14701398(19MB)] 2024/08/21-03:02:47.993038 140539937289792 [top_zip_table.cc:59] core git_version_hash_info_is:commit 7517bad5c1a147474c3830831b9a22c996321ae8 Author: leipeng Date: 2024-01-08 19:18:17 +0800 vfork_cmd: call close before wait_finish g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/08/21-03:02:47.993068 140539937289792 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 7517bad5c1a147474c3830831b9a22c996321ae8 Author: leipeng Date: 2024-01-08 19:18:17 +0800 vfork_cmd: call close before wait_finish g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/08/21-03:02:47.993086 140539937289792 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit 7517bad5c1a147474c3830831b9a22c996321ae8 Author: leipeng Date: 2024-01-08 19:18:17 +0800 vfork_cmd: call close before wait_finish g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/08/21-03:02:47.993099 140539937289792 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 7f80399bfbd2071b6e0f5683e8ab7225b8e6e394 Author: leipeng Date: 2024-01-19 14:52:30 +0800 Change ToplingZipTableOptions::sampleRatio to vector g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/08/21-03:02:47.993111 140539937289792 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit 1240987ca13d0fe562c9b9c037155079f0877840 Author: leipeng Date: 2024-01-19 15:07:04 +0800 Change ToplingZipTableOptions::sampleRatio to vector g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/08/21-03:02:48.647030 140539937289792 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701441.sst) /dev/shm/ToplingTemp/Topling-33511-LIggN7 2024/08/21-03:02:49.111771 140539937289792 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701441.sst): first pass time = 0.46's, 144.415'MB/sec 2024/08/21-03:02:53.639866 140539937289792 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701441.sst): old prealloc_size = 73819750, real_size = 19796848 2024/08/21-03:02:53.640213 140539937289792 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701441.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466386 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.89 raw-val = 0.0541 GB zip-val = 0.0188 GB avg-val = 115.89 avg-zval = 40.31 2024/08/21-03:02:53.640670 140539937289792 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701441.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/21-03:02:54.384381 140539937289792 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701441.sst): second pass time = 4.53's, 11.939'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.74's, 29.050'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2686.424'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.205'MB/sec write SST data time = 0.17's, 124.111'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.543'MB, throughput = 18925.120'MB/sec zip my value time = 4.53's, unzip length = 0.054'GB zip my value throughput = 11.939'MB/sec zip pipeline throughput = 155.293'MB/sec entries = 466386 avg-key = 28.00 avg-zkey = 1.89 avg-val = 115.89 avg-zval = 40.31 usrkeys = 466386 avg-key = 20.00 avg-zkey = 1.89 avg-val = 122.89 avg-zval = 40.31 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466386 bytes = 1399168 } seq expand size = 3264702 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0188 GB dict = 0.54 MB all = 0.0216 GB } UnZip/Zip{ index = 10.5735 value = 2.8753 dict = 1.00 all = 3.1029 } Zip/UnZip{ index = 0.0946 value = 0.3478 dict = 1.00 all = 0.3223 } ---------------------------- total value len = 0.057315 GB avg = 0.123 KB (by entry num) total key len = 0.013059 GB avg = 0.028 KB total ukey len = 0.009328 GB avg = 0.020 KB total ukey num = 0.000466386 Billion total entry num = 0.000466386 Billion write speed all = 0.011596957 MB/sec (with seq num) write speed all = 0.010982106 MB/sec (without seq num) 2024/08/21-03:02:54.420554 140539937289792 [compaction_job.cc:1944] [default] [JOB 62716] Generated table #14701441: 466386 keys, 21627766 bytes, temperature: kWarm 2024/08/21-03:02:54.420668 140539937289792 EVENT_LOG_v1 {"time_micros": 1724180574420600, "cf_name": "default", "job": 62716, "event": "table_file_creation", "file_number": 14701441, "file_size": 21627766, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191359974554, "largest_seqno": 191370407329, "table_properties": {"data_size": 18797920, "index_size": 882176, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13058808, "raw_average_key_size": 28, "raw_value_size": 54050104, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466386, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724180568, "oldest_key_time": 0, "file_creation_time": 1724180568, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14701441, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/21-03:02:54.451270 140539937289792 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701442.sst) /dev/shm/ToplingTemp/Topling-33511-iWOck8 2024/08/21-03:02:55.272139 140539937289792 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701442.sst): first pass time = 0.82's, 81.758'MB/sec 2024/08/21-03:02:56.620642 140539937289792 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701442.sst): old prealloc_size = 73819750, real_size = 19372544 2024/08/21-03:02:56.621546 140539937289792 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701442.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466927 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0184 GB avg-val = 115.72 avg-zval = 39.33 2024/08/21-03:02:56.623537 140539937289792 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701442.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/21-03:02:57.123006 140539937289792 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701442.sst): second pass time = 1.35's, 40.103'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.50's, 42.908'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 1009.564'MB/sec (index lex order gen) rebuild zvType time = 0.42's, 0.278'MB/sec write SST data time = 0.08's, 266.197'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.542'MB, throughput = 20570.605'MB/sec zip my value time = 1.35's, unzip length = 0.054'GB zip my value throughput = 40.103'MB/sec zip pipeline throughput = 96.472'MB/sec entries = 466927 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.72 avg-zval = 39.33 usrkeys = 466927 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.72 avg-zval = 39.33 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466927 bytes = 1750992 } seq expand size = 3268489 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0093 GB value = 0.0540 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0184 GB dict = 0.54 MB all = 0.0216 GB } UnZip/Zip{ index = 10.5005 value = 2.9421 dict = 1.00 all = 3.1134 } Zip/UnZip{ index = 0.0952 value = 0.3399 dict = 1.00 all = 0.3212 } ---------------------------- total value len = 0.114618 GB avg = 0.123 KB (by entry num) total key len = 0.026133 GB avg = 0.028 KB total ukey len = 0.018666 GB avg = 0.020 KB total ukey num = 0.000933313 Billion total entry num = 0.000933313 Billion write speed all = 0.023184087 MB/sec (with seq num) write speed all = 0.021954228 MB/sec (without seq num) 2024/08/21-03:02:57.159917 140539937289792 [compaction_job.cc:1944] [default] [JOB 62716] Generated table #14701442: 466927 keys, 21554664 bytes, temperature: kWarm 2024/08/21-03:02:57.159975 140539937289792 EVENT_LOG_v1 {"time_micros": 1724180577159943, "cf_name": "default", "job": 62716, "event": "table_file_creation", "file_number": 14701442, "file_size": 21554664, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 190586022308, "largest_seqno": 191370405705, "table_properties": {"data_size": 18366256, "index_size": 889344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13073956, "raw_average_key_size": 28, "raw_value_size": 54035034, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466927, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724180574, "oldest_key_time": 0, "file_creation_time": 1724180574, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14701442, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/21-03:02:57.180789 140539937289792 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701443.sst) /dev/shm/ToplingTemp/Topling-33511-5UJi98 2024/08/21-03:02:57.625005 140539937289792 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701443.sst): first pass time = 0.44's, 151.083'MB/sec 2024/08/21-03:02:58.318531 140539937289792 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701443.sst): old prealloc_size = 73819750, real_size = 19401632 2024/08/21-03:02:58.318822 140539937289792 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701443.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466096 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0541 GB zip-val = 0.0184 GB avg-val = 115.98 avg-zval = 39.46 2024/08/21-03:02:58.319179 140539937289792 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701443.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/21-03:02:58.915558 140539937289792 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701443.sst): second pass time = 0.69's, 78.033'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.60's, 35.659'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3386.524'MB/sec (index lex order gen) rebuild zvType time = 0.45's, 0.261'MB/sec write SST data time = 0.15's, 142.373'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.543'MB, throughput = 20346.187'MB/sec zip my value time = 0.69's, unzip length = 0.054'GB zip my value throughput = 78.033'MB/sec zip pipeline throughput = 78.033'MB/sec entries = 466096 avg-key = 28.00 avg-zkey = 1.92 avg-val = 115.98 avg-zval = 39.46 usrkeys = 466096 avg-key = 20.00 avg-zkey = 1.92 avg-val = 122.98 avg-zval = 39.46 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466096 bytes = 1456560 } seq expand size = 3262672 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0184 GB dict = 0.54 MB all = 0.0213 GB } UnZip/Zip{ index = 10.4330 value = 2.9393 dict = 1.00 all = 3.1524 } Zip/UnZip{ index = 0.0958 value = 0.3402 dict = 1.00 all = 0.3172 } ---------------------------- total value len = 0.171939 GB avg = 0.123 KB (by entry num) total key len = 0.039183 GB avg = 0.028 KB total ukey len = 0.027988 GB avg = 0.020 KB total ukey num = 0.001399409 Billion total entry num = 0.001399409 Billion write speed all = 0.034765213 MB/sec (with seq num) write speed all = 0.032921707 MB/sec (without seq num) 2024/08/21-03:02:58.948527 140539937289792 [compaction_job.cc:1944] [default] [JOB 62716] Generated table #14701443: 466096 keys, 21288106 bytes, temperature: kWarm 2024/08/21-03:02:58.948637 140539937289792 EVENT_LOG_v1 {"time_micros": 1724180578948571, "cf_name": "default", "job": 62716, "event": "table_file_creation", "file_number": 14701443, "file_size": 21288106, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191349508980, "largest_seqno": 191370406322, "table_properties": {"data_size": 18391440, "index_size": 893504, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13050688, "raw_average_key_size": 28, "raw_value_size": 54058220, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466096, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724180577, "oldest_key_time": 0, "file_creation_time": 1724180577, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14701443, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/21-03:02:58.987075 140539937289792 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701444.sst) /dev/shm/ToplingTemp/Topling-33511-KeTPRa 2024/08/21-03:02:59.462838 140539937289792 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701444.sst): first pass time = 0.48's, 141.071'MB/sec 2024/08/21-03:03:01.138680 140539937289792 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701444.sst): old prealloc_size = 73819750, real_size = 19011136 2024/08/21-03:03:01.138998 140539937289792 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701444.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466359 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0541 GB zip-val = 0.0180 GB avg-val = 115.90 avg-zval = 38.61 2024/08/21-03:03:01.139429 140539937289792 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701444.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/21-03:03:01.558061 140539937289792 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701444.sst): second pass time = 1.67's, 32.271'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.42's, 49.977'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3018.142'MB/sec (index lex order gen) rebuild zvType time = 0.35's, 0.336'MB/sec write SST data time = 0.07's, 291.152'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.543'MB, throughput = 8544.327'MB/sec zip my value time = 1.67's, unzip length = 0.054'GB zip my value throughput = 32.271'MB/sec zip pipeline throughput = 187.889'MB/sec entries = 466359 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.90 avg-zval = 38.61 usrkeys = 466359 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.90 avg-zval = 38.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466359 bytes = 1515680 } seq expand size = 3264513 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0180 GB dict = 0.54 MB all = 0.0210 GB } UnZip/Zip{ index = 10.5028 value = 3.0018 dict = 1.00 all = 3.2020 } Zip/UnZip{ index = 0.0952 value = 0.3331 dict = 1.00 all = 0.3123 } ---------------------------- total value len = 0.229255 GB avg = 0.123 KB (by entry num) total key len = 0.052242 GB avg = 0.028 KB total ukey len = 0.037315 GB avg = 0.020 KB total ukey num = 0.001865768 Billion total entry num = 0.001865768 Billion write speed all = 0.046333334 MB/sec (with seq num) write speed all = 0.043876541 MB/sec (without seq num) 2024/08/21-03:03:01.587594 140539937289792 [compaction_job.cc:1944] [default] [JOB 62716] Generated table #14701444: 466359 keys, 20958530 bytes, temperature: kWarm 2024/08/21-03:03:01.587661 140539937289792 EVENT_LOG_v1 {"time_micros": 1724180581587626, "cf_name": "default", "job": 62716, "event": "table_file_creation", "file_number": 14701444, "file_size": 20958530, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191306012714, "largest_seqno": 191370405565, "table_properties": {"data_size": 18006320, "index_size": 888064, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13058052, "raw_average_key_size": 28, "raw_value_size": 54050922, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466359, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724180578, "oldest_key_time": 0, "file_creation_time": 1724180578, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14701444, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/21-03:03:01.610865 140539937289792 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701445.sst) /dev/shm/ToplingTemp/Topling-33511-NyQzm9 2024/08/21-03:03:02.055524 140539937289792 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701445.sst): first pass time = 0.44's, 150.935'MB/sec 2024/08/21-03:03:03.914892 140539937289792 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701445.sst): old prealloc_size = 73819750, real_size = 19022384 2024/08/21-03:03:03.915194 140539937289792 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701445.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 466845 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0180 GB avg-val = 115.75 avg-zval = 38.59 2024/08/21-03:03:03.915643 140539937289792 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701445.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/21-03:03:04.345572 140539937289792 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701445.sst): second pass time = 1.86's, 29.077'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.43's, 48.829'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3285.928'MB/sec (index lex order gen) rebuild zvType time = 0.37's, 0.318'MB/sec write SST data time = 0.06's, 332.155'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.542'MB, throughput = 9286.348'MB/sec zip my value time = 1.86's, unzip length = 0.054'GB zip my value throughput = 29.077'MB/sec zip pipeline throughput = 151.691'MB/sec entries = 466845 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.75 avg-zval = 38.59 usrkeys = 466845 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.75 avg-zval = 38.59 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 466845 bytes = 1575616 } seq expand size = 3267915 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0093 GB value = 0.0540 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0180 GB dict = 0.54 MB all = 0.0210 GB } UnZip/Zip{ index = 10.5039 value = 2.9993 dict = 1.00 all = 3.1913 } Zip/UnZip{ index = 0.0952 value = 0.3334 dict = 1.00 all = 0.3134 } ---------------------------- total value len = 0.286560 GB avg = 0.123 KB (by entry num) total key len = 0.065313 GB avg = 0.028 KB total ukey len = 0.046652 GB avg = 0.020 KB total ukey num = 0.002332613 Billion total entry num = 0.002332613 Billion write speed all = 0.057890558 MB/sec (with seq num) write speed all = 0.054820444 MB/sec (without seq num) 2024/08/21-03:03:04.381763 140539937289792 [compaction_job.cc:1944] [default] [JOB 62716] Generated table #14701445: 466845 keys, 21029004 bytes, temperature: kWarm 2024/08/21-03:03:04.381835 140539937289792 EVENT_LOG_v1 {"time_micros": 1724180584381796, "cf_name": "default", "job": 62716, "event": "table_file_creation", "file_number": 14701445, "file_size": 21029004, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191279770607, "largest_seqno": 191370405726, "table_properties": {"data_size": 18016608, "index_size": 888896, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13071660, "raw_average_key_size": 28, "raw_value_size": 54037206, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 466845, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724180581, "oldest_key_time": 0, "file_creation_time": 1724180581, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14701445, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/21-03:03:04.402928 140539937289792 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701446.sst) /dev/shm/ToplingTemp/Topling-33511-IzERa7 2024/08/21-03:03:04.811791 140539937289792 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701446.sst): first pass time = 0.41's, 164.151'MB/sec 2024/08/21-03:03:05.523600 140539937289792 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701446.sst): old prealloc_size = 73819750, real_size = 18629600 2024/08/21-03:03:05.523924 140539937289792 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701446.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 465966 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.0541 GB zip-val = 0.0176 GB avg-val = 116.02 avg-zval = 37.81 2024/08/21-03:03:05.524354 140539937289792 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701446.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/21-03:03:05.929813 140539937289792 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701446.sst): second pass time = 0.71's, 76.057'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.41's, 50.793'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3005.459'MB/sec (index lex order gen) rebuild zvType time = 0.34's, 0.340'MB/sec write SST data time = 0.06's, 327.980'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.543'MB, throughput = 13849.445'MB/sec zip my value time = 0.71's, unzip length = 0.054'GB zip my value throughput = 76.057'MB/sec zip pipeline throughput = 76.057'MB/sec entries = 465966 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.02 avg-zval = 37.81 usrkeys = 465966 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.02 avg-zval = 37.81 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 465966 bytes = 1572656 } seq expand size = 3261762 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0093 GB value = 0.0541 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0176 GB dict = 0.54 MB all = 0.0206 GB } UnZip/Zip{ index = 10.4368 value = 3.0682 dict = 1.00 all = 3.2526 } Zip/UnZip{ index = 0.0958 value = 0.3259 dict = 1.00 all = 0.3074 } ---------------------------- total value len = 0.343883 GB avg = 0.123 KB (by entry num) total key len = 0.078360 GB avg = 0.028 KB total ukey len = 0.055972 GB avg = 0.020 KB total ukey num = 0.002798579 Billion total entry num = 0.002798579 Billion write speed all = 0.069449919 MB/sec (with seq num) write speed all = 0.065767474 MB/sec (without seq num) 2024/08/21-03:03:05.963074 140539937289792 [compaction_job.cc:1944] [default] [JOB 62716] Generated table #14701446: 465966 keys, 20632338 bytes, temperature: kWarm 2024/08/21-03:03:05.963126 140539937289792 EVENT_LOG_v1 {"time_micros": 1724180585963097, "cf_name": "default", "job": 62716, "event": "table_file_creation", "file_number": 14701446, "file_size": 20632338, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191271463728, "largest_seqno": 191370405876, "table_properties": {"data_size": 17619984, "index_size": 892928, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13047048, "raw_average_key_size": 28, "raw_value_size": 54061832, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 465966, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724180584, "oldest_key_time": 0, "file_creation_time": 1724180584, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14701446, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/21-03:03:05.983177 140539937289792 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701447.sst) /dev/shm/ToplingTemp/Topling-33511-bM7Aha 2024/08/21-03:03:06.392455 140539937289792 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701447.sst): first pass time = 0.41's, 163.981'MB/sec 2024/08/21-03:03:07.091251 140539937289792 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701447.sst): old prealloc_size = 73819750, real_size = 19065168 2024/08/21-03:03:07.091597 140539937289792 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701447.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 467558 prefix = 4 raw-key = 0.0075 GB zip-key = 0.0009 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0540 GB zip-val = 0.0181 GB avg-val = 115.53 avg-zval = 38.62 2024/08/21-03:03:07.092017 140539937289792 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701447.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0011 GB, waited 0.000 sec, Key+Value = 0.067 GB 2024/08/21-03:03:07.492759 140539937289792 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701447.sst): second pass time = 0.70's, 77.385'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.40's, 52.486'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2829.311'MB/sec (index lex order gen) rebuild zvType time = 0.34's, 0.347'MB/sec write SST data time = 0.06's, 327.486'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.542'MB, throughput = 10660.018'MB/sec zip my value time = 0.70's, unzip length = 0.054'GB zip my value throughput = 77.385'MB/sec zip pipeline throughput = 77.385'MB/sec entries = 467558 avg-key = 28.00 avg-zkey = 1.90 avg-val = 115.53 avg-zval = 38.62 usrkeys = 467558 avg-key = 20.00 avg-zkey = 1.90 avg-val = 122.53 avg-zval = 38.62 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 467558 bytes = 1578016 } seq expand size = 3272906 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0094 GB value = 0.0540 GB dict = 0.54 MB all = 0.0671 GB } __ZipSize{ index = 0.0009 GB value = 0.0181 GB dict = 0.54 MB all = 0.0211 GB } UnZip/Zip{ index = 10.5041 value = 2.9913 dict = 1.00 all = 3.1845 } Zip/UnZip{ index = 0.0952 value = 0.3343 dict = 1.00 all = 0.3140 } ---------------------------- total value len = 0.401174 GB avg = 0.123 KB (by entry num) total key len = 0.091452 GB avg = 0.028 KB total ukey len = 0.065323 GB avg = 0.020 KB total ukey num = 0.003266137 Billion total entry num = 0.003266137 Billion write speed all = 0.081005374 MB/sec (with seq num) write speed all = 0.076708809 MB/sec (without seq num) 2024/08/21-03:03:07.523456 140539937289792 [compaction_job.cc:1944] [default] [JOB 62716] Generated table #14701447: 467558 keys, 21073566 bytes, temperature: kWarm 2024/08/21-03:03:07.523530 140539937289792 EVENT_LOG_v1 {"time_micros": 1724180587523484, "cf_name": "default", "job": 62716, "event": "table_file_creation", "file_number": 14701447, "file_size": 21073566, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191248629892, "largest_seqno": 191370406981, "table_properties": {"data_size": 18057856, "index_size": 890240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 13091624, "raw_average_key_size": 28, "raw_value_size": 54017260, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 467558, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724180585, "oldest_key_time": 0, "file_creation_time": 1724180585, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14701447, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/21-03:03:07.545743 140539937289792 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701448.sst) /dev/shm/ToplingTemp/Topling-33511-p8vOU9 2024/08/21-03:03:07.682046 140539937289792 [top_zip_table_builder.cc:1344] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701448.sst): first pass time = 0.14's, 245.498'MB/sec 2024/08/21-03:03:07.996143 140539937289792 [top_zip_table_builder.cc:2764] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701448.sst): old prealloc_size = 73819750, real_size = 9497104 2024/08/21-03:03:07.996338 140539937289792 [top_zip_table_builder.cc:1765] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701448.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 232726 prefix = 4 raw-key = 0.0037 GB zip-key = 0.0004 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0269 GB zip-val = 0.0090 GB avg-val = 115.75 avg-zval = 38.65 2024/08/21-03:03:07.996573 140539937289792 [top_zip_table_builder.cc:1295] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701448.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.000 GB, reorder workingMem = 0.0005 GB, waited 0.000 sec, Key+Value = 0.033 GB 2024/08/21-03:03:08.201844 140539937289792 [top_zip_table_builder.cc:3092] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00/14701448.sst): second pass time = 0.31's, 85.957'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.21's, 50.927'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2560.901'MB/sec (index lex order gen) rebuild zvType time = 0.17's, 0.338'MB/sec write SST data time = 0.03's, 314.842'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.277'MB, throughput = 10544.568'MB/sec zip my value time = 0.31's, unzip length = 0.027'GB zip my value throughput = 85.957'MB/sec zip pipeline throughput = 85.957'MB/sec entries = 232726 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.75 avg-zval = 38.65 usrkeys = 232726 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.75 avg-zval = 38.65 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 232726 bytes = 756368 } seq expand size = 1629082 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0047 GB value = 0.0269 GB dict = 0.28 MB all = 0.0335 GB } __ZipSize{ index = 0.0004 GB value = 0.0090 GB dict = 0.28 MB all = 0.0105 GB } UnZip/Zip{ index = 10.4809 value = 2.9949 dict = 1.00 all = 3.1939 } Zip/UnZip{ index = 0.0954 value = 0.3339 dict = 1.00 all = 0.3131 } ---------------------------- total value len = 0.429741 GB avg = 0.123 KB (by entry num) total key len = 0.097968 GB avg = 0.028 KB total ukey len = 0.069977 GB avg = 0.020 KB total ukey num = 0.003498863 Billion total entry num = 0.003498863 Billion write speed all = 0.086764296 MB/sec (with seq num) write speed all = 0.082162119 MB/sec (without seq num) 2024/08/21-03:03:08.219194 140539937289792 [compaction_job.cc:1944] [default] [JOB 62716] Generated table #14701448: 232726 keys, 10474580 bytes, temperature: kWarm 2024/08/21-03:03:08.219255 140539937289792 EVENT_LOG_v1 {"time_micros": 1724180588219219, "cf_name": "default", "job": 62716, "event": "table_file_creation", "file_number": 14701448, "file_size": 10474580, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 191325164752, "largest_seqno": 191368731083, "table_properties": {"data_size": 8994624, "index_size": 444096, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 6516328, "raw_average_key_size": 28, "raw_value_size": 26938372, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 232726, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1724180587, "oldest_key_time": 0, "file_creation_time": 1724180587, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "a007a635-5c0a-4b4a-be96-b9c695cb016b", "db_session_id": "90ZT1ZM5QNL093I8HHQG", "orig_file_number": 14701448, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/08/21-03:03:08.230617 140539937289792 [compaction_job.cc:699] job-62716: subcompact[0], size: 0.158639 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-62716/att-00 [14701441,14701442,14701443,14701444,14701445,14701446,14701447,14701448] 2024/08/21-03:03:08.267024 140539937289792 (Original Log Time 2024/08/21-03:02:47.548587) EVENT_LOG_v1 {"time_micros": 1724180567548554, "job": 62716, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L1": [14701420], "files_L2": [14701382, 14701383, 14701384, 14701385, 14701386, 14701397, 14701398], "score": 0.997776, "input_data_size": 181806667, "oldest_snapshot_seqno": 191370424993} 2024/08/21-03:03:08.267028 140539937289792 (Original Log Time 2024/08/21-03:03:08.266210) [compaction_job.cc:2018] [default] [JOB 62716] Compacted 1@1 + 7@2 files to L2 => 158638554 bytes 2024/08/21-03:03:08.267029 140539937289792 (Original Log Time 2024/08/21-03:03:08.266973) [compaction_job.cc:1220] [default] compacted to: files[0 0 8 0 0 0 0] max score 0.06, MB/sec: 8.8 rd, 7.7 wr, level 2, files in(1, 7) out(8 +0 blob) MB in(31.8, 141.6 +0.0 blob) out(151.3 +0.0 blob), read-write-amplify(10.2) write-amplify(4.8) OK, records in: 3498863, records dropped: 0 output_compression: Snappy 2024/08/21-03:03:08.267034 140539937289792 (Original Log Time 2024/08/21-03:03:08.267011) EVENT_LOG_v1 {"time_micros": 1724180588266994, "cf": "default", "job": 62716, "event": "compaction_finished", "compaction_time_micros": 20682034, "compaction_time_cpu_micros": 6020097, "output_level": 2, "num_output_files": 8, "total_output_size": 158638554, "num_input_records": 3498863, "num_output_records": 3498863, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 178056992, "file_range_sync_nanos": 0, "file_fsync_nanos": 6660, "file_prepare_write_nanos": 20825, "lsm_state": [0, 0, 8, 0, 0, 0, 0]} 2024/08/21-03:03:08.393969 140539937289792 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"a007a635-5c0a-4b4a-be96-b9c695cb016b","dbStarts":1724092920,"starts":1724180567,"executesMs":20832,"compactionJobId":62716,"attempt":0,"compactionInputRawBytes":503217114,"compactionInputZipBytes":181806667,"compactionOutputRawBytes":0,"compactionOutputZipBytes":158638554,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-f4043880e46b8d25a739de445ef7b33b-416b9108cc4ee720-00"}}, response = 2024/08/21-03:03:08.394041 140539937289792 INFO dcompact_worker.cpp:1456: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-08-20T02.42.00/.rocksdb/job-62716/att-00: olev 2, work 20.833 s, result 6.381 ms, install 0.839 ms, input{raw 479.905 MiB zip 173.384 MiB}