2024/11/20-16:36:09.138129 140257030129216 DEBG dcompact_worker.cpp:2219: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00: fork to child time = 0.009117 sec 2024/11/20-16:36:09.138540 140257030129216 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00 2024/11/20-16:36:09.144847 140257030129216 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00 2024/11/20-16:36:09.146538 140257030129216 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/20-16:36:09.146738 140257030129216 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/20-16:36:09.194952 140257030129216 [version_set.cc:6211] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16186/att-00/MANIFEST-019309 2024/11/20-16:36:09.250298 140257030129216 [version_set.cc:6254] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16186/att-00/MANIFEST-019309 succeeded,manifest_file_number is 19309, next_file_number is 46091, last_sequence is 784244359, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 45861 2024/11/20-16:36:09.250304 140257030129216 [version_set.cc:6269] Column family [default] (ID 0), log number is 0 2024/11/20-16:36:09.250388 140257030129216 [version_set.cc:5724] Creating manifest 46091 2024/11/20-16:36:09.283107 140257030129216 DEBG dcompact_worker.cpp:1225: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/20-16:36:09.303032 140257030129216 [compaction_job.cc:2437] [default] [JOB 16186] Compacting 1@2 + 17@3 files to L3, score 1.00, subcompactions 1 : 1 2024/11/20-16:36:09.303057 140257030129216 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 12497 Base level 2, inputs: [46020(27MB)], [46080(36MB) 46081(6458KB) 45973(37MB) 45974(36MB) 45975(37MB) 45976(36MB) 45977(37MB) 45978(35MB) 45979(35MB) 45980(37MB) 45981(38MB) 45982(37MB) 45983(36MB) 45984(36MB) 45985(37MB) 45986(37MB) 46083(37MB)] 2024/11/20-16:36:09.309104 140257030129216 [top_zip_table.cc:59] core git_version_hash_info_is:commit 81964c21ebbcc178d8e99dae0fbf61bded2729cc Author: rockeet Date: Mon Nov 11 21:28:46 2024 +0800 Makefile: fix gen git-version-*.cpp GIT_PATH_ARG = :!src/terark/fsa :!tools/fsa :!src/terark/zbs :!tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:36:09.309122 140257030129216 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit 5dfecb7d107196f044ff5bf78dc39d2e725b9d9b Author: leipeng Date: Thu Nov 7 15:22:45 2024 +0800 Makefile: fix for x86_64 CPU older than haswell GIT_PATH_ARG = src/terark/fsa tools/fsa g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:36:09.309137 140257030129216 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit d44dd5ff72707128fe3c916077b43d2093c01f7c Author: leipeng Date: Sun Oct 27 23:38:26 2024 +0800 Fix for msvc, when toplingdb on msvc GIT_PATH_ARG = src/terark/zbs tools/zbs g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/zstd -Iboost-include DEFS = -DTOPLING_IO_WITH_URING=1 -DDIVSUFSORT_API= CXXFLAGS = -fPIC -time -fdiagnostics-color -mcx16 -fno-stack-protector WARNINGS = -Wno-class-memaccess -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -Wno-deprecated-declarations -Wstrict-aliasing=3 RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 LDFLAGS = -fPIC -rdynamic 2024/11/20-16:36:09.309143 140257030129216 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit 79614a7e7dc27362fc5151b0797eb7d83c674c7b Author: rockeet Date: Sun Nov 10 15:02:21 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:36:09.309151 140257030129216 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit c46b65a52d9c74aa8b87aeb8438620ff2dbc669c Author: rockeet Date: Sun Nov 10 15:02:42 2024 +0800 cpu_has_bmi2.sh: detect by compile c++ code check __BMI2__ g++ (GCC) 12.2.1 20221121 (Red Hat 12.2.1-7) Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/20-16:36:09.836805 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046092.sst) /dev/shm/ToplingTemp/Topling-589226-Eukj3J 2024/11/20-16:36:10.332269 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046092.sst): first pass time = 0.50's, 270.916'MB/sec 2024/11/20-16:36:11.489215 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046092.sst): old prealloc_size = 147639500, real_size = 34600208 2024/11/20-16:36:11.489724 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046092.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926740 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0326 GB avg-val = 116.83 avg-zval = 35.18 2024/11/20-16:36:11.490539 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046092.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:12.327164 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046092.sst): second pass time = 1.16's, 93.650'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.183'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3592.549'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.324'MB/sec write SST data time = 0.12's, 319.067'MB/sec dict compress time = 6517941678.37's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 31490.760'MB/sec zip my value time = 1.16's, unzip length = 0.108'GB zip my value throughput = 93.650'MB/sec zip pipeline throughput = 93.650'MB/sec entries = 926740 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.83 avg-zval = 35.18 usrkeys = 926740 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.83 avg-zval = 35.18 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926740 bytes = 3243600 } seq expand size = 6487180 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0326 GB dict = 1.08 MB all = 0.0387 GB } UnZip/Zip{ index = 10.5109 value = 3.3206 dict = 1.00 all = 3.4683 } Zip/UnZip{ index = 0.0951 value = 0.3011 dict = 1.00 all = 0.2883 } ---------------------------- total value len = 2183.597536 GB avg = 0.300 KB (by entry num) total key len = 191.589241 GB avg = 0.026 KB total ukey len = 134.138866 GB avg = 0.014 KB total ukey num = 9.327058806 Billion total entry num = 7.271208952 Billion write speed all = 150.184269546 MB/sec (with seq num) write speed all = 146.506171585 MB/sec (without seq num) 2024/11/20-16:36:12.373196 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46092: 926740 keys, 38698258 bytes, temperature: kWarm 2024/11/20-16:36:12.373273 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091772373235, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46092, "file_size": 38698258, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 644972643, "largest_seqno": 783153083, "table_properties": {"data_size": 32604928, "index_size": 1763392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25948720, "raw_average_key_size": 28, "raw_value_size": 108269008, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926740, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091769, "oldest_key_time": 0, "file_creation_time": 1732091769, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46092, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:12.406644 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046093.sst) /dev/shm/ToplingTemp/Topling-589226-qAt96H 2024/11/20-16:36:13.266525 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046093.sst): first pass time = 0.86's, 156.094'MB/sec 2024/11/20-16:36:14.406602 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046093.sst): old prealloc_size = 147639500, real_size = 35850960 2024/11/20-16:36:14.407168 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046093.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923887 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0338 GB avg-val = 117.28 avg-zval = 36.61 2024/11/20-16:36:14.407918 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046093.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:15.192292 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046093.sst): second pass time = 1.14's, 95.111'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.79's, 50.624'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3364.324'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.350'MB/sec write SST data time = 0.12's, 319.092'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 19642.041'MB/sec zip my value time = 1.14's, unzip length = 0.108'GB zip my value throughput = 95.111'MB/sec zip pipeline throughput = 95.111'MB/sec entries = 923887 avg-key = 28.00 avg-zkey = 1.94 avg-val = 117.28 avg-zval = 36.61 usrkeys = 923887 avg-key = 20.00 avg-zkey = 1.94 avg-val = 123.37 avg-zval = 36.61 TagRS{ kinds = RS_Key0_TagN bytes = 144440 } TagArray{ size = 804925 bytes = 2917872 } seq expand size = 5634475 multi value expand size = 0 cnt WriteAppend = 35 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0338 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.3083 value = 3.2030 dict = 1.00 all = 3.3745 } Zip/UnZip{ index = 0.0970 value = 0.3122 dict = 1.00 all = 0.2963 } ---------------------------- total value len = 2183.711519 GB avg = 0.300 KB (by entry num) total key len = 191.615110 GB avg = 0.026 KB total ukey len = 134.157344 GB avg = 0.014 KB total ukey num = 9.327982693 Billion total entry num = 7.272132839 Billion write speed all = 150.165907879 MB/sec (with seq num) write speed all = 146.488008876 MB/sec (without seq num) 2024/11/20-16:36:15.240550 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46093: 923887 keys, 39774486 bytes, temperature: kWarm 2024/11/20-16:36:15.240612 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091775240582, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46093, "file_size": 39774486, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 783299150, "table_properties": {"data_size": 33827264, "index_size": 1792512, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25868836, "raw_average_key_size": 28, "raw_value_size": 108348902, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923887, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091772, "oldest_key_time": 0, "file_creation_time": 1732091772, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46093, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:15.282865 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046094.sst) /dev/shm/ToplingTemp/Topling-589226-ivqQgG 2024/11/20-16:36:16.098178 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046094.sst): first pass time = 0.82's, 164.630'MB/sec 2024/11/20-16:36:17.220794 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046094.sst): old prealloc_size = 147639500, real_size = 34731936 2024/11/20-16:36:17.221380 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046094.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924174 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 117.23 avg-zval = 35.43 2024/11/20-16:36:17.222172 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046094.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:18.057093 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046094.sst): second pass time = 1.12's, 96.587'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.424'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3208.479'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.324'MB/sec write SST data time = 0.12's, 315.324'MB/sec dict compress time = 140278.11's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 18255.295'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.587'MB/sec zip pipeline throughput = 96.587'MB/sec entries = 924174 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.23 avg-zval = 35.43 usrkeys = 924174 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.23 avg-zval = 35.43 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924174 bytes = 3234624 } seq expand size = 6469218 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5131 value = 3.3089 dict = 1.00 all = 3.4572 } Zip/UnZip{ index = 0.0951 value = 0.3022 dict = 1.00 all = 0.2893 } ---------------------------- total value len = 2183.826329 GB avg = 0.300 KB (by entry num) total key len = 191.640987 GB avg = 0.026 KB total ukey len = 134.175828 GB avg = 0.014 KB total ukey num = 9.328906867 Billion total entry num = 7.273057013 Billion write speed all = 150.147608957 MB/sec (with seq num) write speed all = 146.469908615 MB/sec (without seq num) 2024/11/20-16:36:18.106473 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46094: 924174 keys, 38823082 bytes, temperature: kWarm 2024/11/20-16:36:18.106543 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091778106512, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46094, "file_size": 38823082, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 635932289, "largest_seqno": 783301130, "table_properties": {"data_size": 32742544, "index_size": 1758144, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25876872, "raw_average_key_size": 28, "raw_value_size": 108340956, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924174, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091775, "oldest_key_time": 0, "file_creation_time": 1732091775, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46094, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:18.142816 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046095.sst) /dev/shm/ToplingTemp/Topling-589226-YZRC5F 2024/11/20-16:36:18.958311 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046095.sst): first pass time = 0.82's, 164.594'MB/sec 2024/11/20-16:36:20.082353 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046095.sst): old prealloc_size = 147639500, real_size = 34701792 2024/11/20-16:36:20.082923 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046095.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924677 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0327 GB avg-val = 117.15 avg-zval = 35.38 2024/11/20-16:36:20.083762 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046095.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:20.921804 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046095.sst): second pass time = 1.12's, 96.453'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 46.214'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3298.715'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.324'MB/sec write SST data time = 0.12's, 310.978'MB/sec dict compress time = 140278.11's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31776.158'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.453'MB/sec zip pipeline throughput = 96.453'MB/sec entries = 924677 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.15 avg-zval = 35.38 usrkeys = 924677 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.15 avg-zval = 35.38 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924677 bytes = 3236384 } seq expand size = 6472739 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0327 GB dict = 1.08 MB all = 0.0388 GB } UnZip/Zip{ index = 10.5134 value = 3.3116 dict = 1.00 all = 3.4598 } Zip/UnZip{ index = 0.0951 value = 0.3020 dict = 1.00 all = 0.2890 } ---------------------------- total value len = 2183.941129 GB avg = 0.300 KB (by entry num) total key len = 191.666878 GB avg = 0.026 KB total ukey len = 134.194321 GB avg = 0.014 KB total ukey num = 9.329831544 Billion total entry num = 7.273981690 Billion write speed all = 150.129317341 MB/sec (with seq num) write speed all = 146.451815321 MB/sec (without seq num) 2024/11/20-16:36:20.970519 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46095: 924677 keys, 38793936 bytes, temperature: kWarm 2024/11/20-16:36:20.970587 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091780970557, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46095, "file_size": 38793936, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 635932320, "largest_seqno": 783301282, "table_properties": {"data_size": 32711376, "index_size": 1759040, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25890956, "raw_average_key_size": 28, "raw_value_size": 108326850, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924677, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091778, "oldest_key_time": 0, "file_creation_time": 1732091778, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46095, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:21.004624 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046096.sst) /dev/shm/ToplingTemp/Topling-589226-8b0ybH 2024/11/20-16:36:21.812579 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046096.sst): first pass time = 0.81's, 166.129'MB/sec 2024/11/20-16:36:22.932234 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046096.sst): old prealloc_size = 147639500, real_size = 35020272 2024/11/20-16:36:22.932742 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046096.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924732 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0330 GB avg-val = 117.14 avg-zval = 35.72 2024/11/20-16:36:22.933549 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046096.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:23.762935 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046096.sst): second pass time = 1.12's, 96.826'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.085'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3638.497'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.327'MB/sec write SST data time = 0.12's, 318.554'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 31929.230'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.826'MB/sec zip pipeline throughput = 96.826'MB/sec entries = 924732 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.14 avg-zval = 35.72 usrkeys = 924732 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.14 avg-zval = 35.72 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924732 bytes = 3236576 } seq expand size = 6473124 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0330 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.5137 value = 3.2796 dict = 1.00 all = 3.4316 } Zip/UnZip{ index = 0.0951 value = 0.3049 dict = 1.00 all = 0.2914 } ---------------------------- total value len = 2184.055927 GB avg = 0.300 KB (by entry num) total key len = 191.692770 GB avg = 0.026 KB total ukey len = 134.212816 GB avg = 0.014 KB total ukey num = 9.330756276 Billion total entry num = 7.274906422 Billion write speed all = 150.111256282 MB/sec (with seq num) write speed all = 146.433946999 MB/sec (without seq num) 2024/11/20-16:36:23.811262 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46096: 924732 keys, 39112500 bytes, temperature: kWarm 2024/11/20-16:36:23.811331 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091783811300, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46096, "file_size": 39112500, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 635932374, "largest_seqno": 783299744, "table_properties": {"data_size": 33029792, "index_size": 1759104, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25892496, "raw_average_key_size": 28, "raw_value_size": 108325304, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924732, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091780, "oldest_key_time": 0, "file_creation_time": 1732091780, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46096, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:23.844319 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046097.sst) /dev/shm/ToplingTemp/Topling-589226-azNlJJ 2024/11/20-16:36:24.655350 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046097.sst): first pass time = 0.81's, 165.500'MB/sec 2024/11/20-16:36:25.872671 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046097.sst): old prealloc_size = 147639500, real_size = 36023360 2024/11/20-16:36:25.873275 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046097.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927622 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1082 GB zip-val = 0.0340 GB avg-val = 116.69 avg-zval = 36.62 2024/11/20-16:36:25.874071 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046097.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:26.642302 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046097.sst): second pass time = 1.22's, 88.989'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.77's, 52.113'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3234.155'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.361'MB/sec write SST data time = 0.13's, 315.988'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 17836.600'MB/sec zip my value time = 1.22's, unzip length = 0.108'GB zip my value throughput = 88.989'MB/sec zip pipeline throughput = 88.989'MB/sec entries = 927622 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.69 avg-zval = 36.62 usrkeys = 927622 avg-key = 20.00 avg-zkey = 1.97 avg-val = 123.10 avg-zval = 36.62 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 850051 bytes = 3081456 } seq expand size = 5950357 multi value expand size = 0 cnt WriteAppend = 44 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0401 GB } UnZip/Zip{ index = 10.1774 value = 3.1866 dict = 1.00 all = 3.3464 } Zip/UnZip{ index = 0.0983 value = 0.3138 dict = 1.00 all = 0.2988 } ---------------------------- total value len = 2184.170122 GB avg = 0.300 KB (by entry num) total key len = 191.718744 GB avg = 0.026 KB total ukey len = 134.231368 GB avg = 0.014 KB total ukey num = 9.331683898 Billion total entry num = 7.275834044 Billion write speed all = 150.092805986 MB/sec (with seq num) write speed all = 146.415696797 MB/sec (without seq num) 2024/11/20-16:36:26.690397 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46097: 927622 keys, 40107838 bytes, temperature: kWarm 2024/11/20-16:36:26.690464 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091786690435, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46097, "file_size": 40107838, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 783300907, "table_properties": {"data_size": 33968368, "index_size": 1822912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25973416, "raw_average_key_size": 28, "raw_value_size": 108244448, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927622, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091783, "oldest_key_time": 0, "file_creation_time": 1732091783, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46097, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:26.724749 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046098.sst) /dev/shm/ToplingTemp/Topling-589226-ELGM2F 2024/11/20-16:36:27.492702 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046098.sst): first pass time = 0.77's, 174.785'MB/sec 2024/11/20-16:36:28.756232 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046098.sst): old prealloc_size = 147639500, real_size = 36547200 2024/11/20-16:36:28.756773 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046098.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927002 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.95 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.79 avg-zval = 37.22 2024/11/20-16:36:28.756916 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046098.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:29.449247 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046098.sst): second pass time = 1.26's, 85.743'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.69's, 54.214'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3506.045'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.408'MB/sec write SST data time = 0.12's, 300.650'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32552.416'MB/sec zip my value time = 1.26's, unzip length = 0.108'GB zip my value throughput = 85.743'MB/sec zip pipeline throughput = 85.743'MB/sec entries = 927002 avg-key = 28.00 avg-zkey = 1.95 avg-val = 116.79 avg-zval = 37.22 usrkeys = 927002 avg-key = 20.00 avg-zkey = 1.95 avg-val = 116.83 avg-zval = 37.22 TagRS{ kinds = RS_Key0_TagN bytes = 144960 } TagArray{ size = 5910 bytes = 19216 } seq expand size = 41370 multi value expand size = 0 cnt WriteAppend = 65 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.2399 value = 3.1376 dict = 1.00 all = 3.5725 } Zip/UnZip{ index = 0.0977 value = 0.3187 dict = 1.00 all = 0.2799 } ---------------------------- total value len = 2184.278425 GB avg = 0.300 KB (by entry num) total key len = 191.744700 GB avg = 0.026 KB total ukey len = 134.249908 GB avg = 0.014 KB total ukey num = 9.332610900 Billion total entry num = 7.276761046 Billion write speed all = 150.074675732 MB/sec (with seq num) write speed all = 146.397750056 MB/sec (without seq num) 2024/11/20-16:36:29.495259 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46098: 927002 keys, 37569841 bytes, temperature: kWarm 2024/11/20-16:36:29.495327 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091789495295, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46098, "file_size": 37569841, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 538604047, "table_properties": {"data_size": 34504688, "index_size": 1810560, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25956056, "raw_average_key_size": 28, "raw_value_size": 108261720, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927002, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091786, "oldest_key_time": 0, "file_creation_time": 1732091786, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46098, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:29.534018 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046099.sst) /dev/shm/ToplingTemp/Topling-589226-Jb9lfJ 2024/11/20-16:36:30.297162 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046099.sst): first pass time = 0.76's, 175.889'MB/sec 2024/11/20-16:36:31.614429 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046099.sst): old prealloc_size = 147639500, real_size = 36430096 2024/11/20-16:36:31.614990 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046099.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926944 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.94 raw-val = 0.1083 GB zip-val = 0.0344 GB avg-val = 116.80 avg-zval = 37.11 2024/11/20-16:36:31.615136 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046099.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:32.309515 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046099.sst): second pass time = 1.32's, 82.241'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 53.860'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3344.585'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.407'MB/sec write SST data time = 0.13's, 298.663'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33110.982'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.241'MB/sec zip pipeline throughput = 82.241'MB/sec entries = 926944 avg-key = 28.00 avg-zkey = 1.94 avg-val = 116.80 avg-zval = 37.11 usrkeys = 926944 avg-key = 20.00 avg-zkey = 1.94 avg-val = 116.80 avg-zval = 37.11 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 936 bytes = 3056 } seq expand size = 6552 multi value expand size = 0 cnt WriteAppend = 33 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0344 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.3060 value = 3.1472 dict = 1.00 all = 3.5852 } Zip/UnZip{ index = 0.0970 value = 0.3177 dict = 1.00 all = 0.2789 } ---------------------------- total value len = 2184.386695 GB avg = 0.300 KB (by entry num) total key len = 191.770654 GB avg = 0.026 KB total ukey len = 134.268447 GB avg = 0.014 KB total ukey num = 9.333537844 Billion total entry num = 7.277687990 Billion write speed all = 150.056044403 MB/sec (with seq num) write speed all = 146.379314586 MB/sec (without seq num) 2024/11/20-16:36:32.354013 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46099: 926944 keys, 37436295 bytes, temperature: kWarm 2024/11/20-16:36:32.354102 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091792354070, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46099, "file_size": 37436295, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 538604114, "table_properties": {"data_size": 34399360, "index_size": 1798848, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25954432, "raw_average_key_size": 28, "raw_value_size": 108263308, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926944, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091789, "oldest_key_time": 0, "file_creation_time": 1732091789, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46099, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:32.387793 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046100.sst) /dev/shm/ToplingTemp/Topling-589226-Yx8L0I 2024/11/20-16:36:33.176073 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046100.sst): first pass time = 0.79's, 170.274'MB/sec 2024/11/20-16:36:34.462737 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046100.sst): old prealloc_size = 147639500, real_size = 36523984 2024/11/20-16:36:34.463353 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046100.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926765 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 116.82 avg-zval = 37.19 2024/11/20-16:36:34.463816 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046100.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:35.158175 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046100.sst): second pass time = 1.29's, 84.205'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.70's, 56.216'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3131.312'MB/sec (index lex order gen) rebuild zvType time = 0.57's, 0.407'MB/sec write SST data time = 0.13's, 310.798'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33864.949'MB/sec zip my value time = 1.29's, unzip length = 0.108'GB zip my value throughput = 84.205'MB/sec zip pipeline throughput = 84.205'MB/sec entries = 926765 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.82 avg-zval = 37.19 usrkeys = 926765 avg-key = 20.00 avg-zkey = 1.97 avg-val = 120.33 avg-zval = 37.19 TagRS{ kinds = RS_Key0_TagN bytes = 144920 } TagArray{ size = 464596 bytes = 1568032 } seq expand size = 3252172 multi value expand size = 0 cnt WriteAppend = 49 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.1769 value = 3.1409 dict = 1.00 all = 3.4331 } Zip/UnZip{ index = 0.0983 value = 0.3184 dict = 1.00 all = 0.2913 } ---------------------------- total value len = 2184.498216 GB avg = 0.300 KB (by entry num) total key len = 191.796604 GB avg = 0.026 KB total ukey len = 134.286982 GB avg = 0.014 KB total ukey num = 9.334464609 Billion total entry num = 7.278614755 Billion write speed all = 150.037734716 MB/sec (with seq num) write speed all = 146.361198081 MB/sec (without seq num) 2024/11/20-16:36:35.206733 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46100: 926765 keys, 39094993 bytes, temperature: kWarm 2024/11/20-16:36:35.206803 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091795206772, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46100, "file_size": 39094993, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 541395865, "table_properties": {"data_size": 34470784, "index_size": 1821312, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25949420, "raw_average_key_size": 28, "raw_value_size": 108268422, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926765, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091792, "oldest_key_time": 0, "file_creation_time": 1732091792, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46100, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:35.242588 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046101.sst) /dev/shm/ToplingTemp/Topling-589226-67WB6H 2024/11/20-16:36:36.066312 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046101.sst): first pass time = 0.82's, 162.950'MB/sec 2024/11/20-16:36:37.219456 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046101.sst): old prealloc_size = 147639500, real_size = 36972016 2024/11/20-16:36:37.219990 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046101.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927611 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1082 GB zip-val = 0.0350 GB avg-val = 116.69 avg-zval = 37.69 2024/11/20-16:36:37.220733 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046101.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:37.991857 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046101.sst): second pass time = 1.15's, 93.938'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.77's, 52.774'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3464.524'MB/sec (index lex order gen) rebuild zvType time = 0.64's, 0.361'MB/sec write SST data time = 0.13's, 315.667'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 17779.333'MB/sec zip my value time = 1.15's, unzip length = 0.108'GB zip my value throughput = 93.938'MB/sec zip pipeline throughput = 93.938'MB/sec entries = 927611 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.69 avg-zval = 37.69 usrkeys = 927611 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.49 avg-zval = 37.69 TagRS{ kinds = RS_Key0_TagN bytes = 145040 } TagArray{ size = 768806 bytes = 2786944 } seq expand size = 5381642 multi value expand size = 0 cnt WriteAppend = 18 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0350 GB dict = 1.08 MB all = 0.0408 GB } UnZip/Zip{ index = 10.4449 value = 3.0959 dict = 1.00 all = 3.2927 } Zip/UnZip{ index = 0.0957 value = 0.3230 dict = 1.00 all = 0.3037 } ---------------------------- total value len = 2184.611842 GB avg = 0.300 KB (by entry num) total key len = 191.822577 GB avg = 0.026 KB total ukey len = 134.305535 GB avg = 0.014 KB total ukey num = 9.335392220 Billion total entry num = 7.279542366 Billion write speed all = 150.019707864 MB/sec (with seq num) write speed all = 146.343360441 MB/sec (without seq num) 2024/11/20-16:36:38.041943 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46101: 927611 keys, 40761650 bytes, temperature: kWarm 2024/11/20-16:36:38.042011 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091798041979, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46101, "file_size": 40761650, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 783301146, "table_properties": {"data_size": 34963744, "index_size": 1776192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25973108, "raw_average_key_size": 28, "raw_value_size": 108244634, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927611, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091795, "oldest_key_time": 0, "file_creation_time": 1732091795, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46101, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:38.078018 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046102.sst) /dev/shm/ToplingTemp/Topling-589226-7WwhCF 2024/11/20-16:36:38.899102 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046102.sst): first pass time = 0.82's, 163.477'MB/sec 2024/11/20-16:36:40.019154 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046102.sst): old prealloc_size = 147639500, real_size = 35678688 2024/11/20-16:36:40.019702 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046102.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924425 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0337 GB avg-val = 117.19 avg-zval = 36.42 2024/11/20-16:36:40.020514 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046102.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:40.850735 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046102.sst): second pass time = 1.12's, 96.807'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 47.825'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3490.182'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.327'MB/sec write SST data time = 0.12's, 319.121'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 34932.179'MB/sec zip my value time = 1.12's, unzip length = 0.108'GB zip my value throughput = 96.807'MB/sec zip pipeline throughput = 96.807'MB/sec entries = 924425 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.19 avg-zval = 36.42 usrkeys = 924425 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.19 avg-zval = 36.42 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924425 bytes = 3235504 } seq expand size = 6470975 multi value expand size = 0 cnt WriteAppend = 17 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0337 GB dict = 1.08 MB all = 0.0398 GB } UnZip/Zip{ index = 10.3769 value = 3.2179 dict = 1.00 all = 3.3748 } Zip/UnZip{ index = 0.0964 value = 0.3108 dict = 1.00 all = 0.2963 } ---------------------------- total value len = 2184.726647 GB avg = 0.300 KB (by entry num) total key len = 191.848461 GB avg = 0.026 KB total ukey len = 134.324023 GB avg = 0.014 KB total ukey num = 9.336316645 Billion total entry num = 7.280466791 Billion write speed all = 150.001517690 MB/sec (with seq num) write speed all = 146.325366864 MB/sec (without seq num) 2024/11/20-16:36:40.898677 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46102: 924425 keys, 39770488 bytes, temperature: kWarm 2024/11/20-16:36:40.898746 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091800898715, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46102, "file_size": 39770488, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 632970998, "largest_seqno": 783300514, "table_properties": {"data_size": 33665680, "index_size": 1781696, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25883900, "raw_average_key_size": 28, "raw_value_size": 108333934, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924425, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091798, "oldest_key_time": 0, "file_creation_time": 1732091798, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46102, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:40.945736 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046103.sst) /dev/shm/ToplingTemp/Topling-589226-x3aNII 2024/11/20-16:36:41.757438 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046103.sst): first pass time = 0.81's, 165.364'MB/sec 2024/11/20-16:36:42.952317 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046103.sst): old prealloc_size = 147639500, real_size = 35980448 2024/11/20-16:36:42.952991 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046103.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927281 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1083 GB zip-val = 0.0340 GB avg-val = 116.74 avg-zval = 36.62 2024/11/20-16:36:42.953640 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046103.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:43.868431 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046103.sst): second pass time = 1.19's, 90.658'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.92's, 42.862'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2748.134'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.22's, 174.675'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33020.234'MB/sec zip my value time = 1.19's, unzip length = 0.108'GB zip my value throughput = 90.658'MB/sec zip pipeline throughput = 90.658'MB/sec entries = 927281 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.74 avg-zval = 36.62 usrkeys = 927281 avg-key = 20.00 avg-zkey = 1.93 avg-val = 121.50 avg-zval = 36.62 TagRS{ kinds = RS_Key0_TagN bytes = 145000 } TagArray{ size = 629641 bytes = 2282464 } seq expand size = 4407487 multi value expand size = 0 cnt WriteAppend = 23 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0340 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.3758 value = 3.1876 dict = 1.00 all = 3.4182 } Zip/UnZip{ index = 0.0964 value = 0.3137 dict = 1.00 all = 0.2926 } ---------------------------- total value len = 2184.839308 GB avg = 0.300 KB (by entry num) total key len = 191.874424 GB avg = 0.026 KB total ukey len = 134.342569 GB avg = 0.014 KB total ukey num = 9.337243926 Billion total entry num = 7.281394072 Billion write speed all = 149.981700735 MB/sec (with seq num) write speed all = 146.305781833 MB/sec (without seq num) 2024/11/20-16:36:43.917319 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46103: 927281 keys, 39266100 bytes, temperature: kWarm 2024/11/20-16:36:43.917385 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091803917354, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46103, "file_size": 39266100, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 783299907, "table_properties": {"data_size": 33961040, "index_size": 1787392, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25963868, "raw_average_key_size": 28, "raw_value_size": 108253978, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927281, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091800, "oldest_key_time": 0, "file_creation_time": 1732091800, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46103, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:43.963024 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046104.sst) /dev/shm/ToplingTemp/Topling-589226-lgXviJ 2024/11/20-16:36:44.909088 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046104.sst): first pass time = 0.95's, 141.878'MB/sec 2024/11/20-16:36:47.607238 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046104.sst): old prealloc_size = 147639500, real_size = 36530576 2024/11/20-16:36:47.607872 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046104.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924395 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0345 GB avg-val = 117.20 avg-zval = 37.37 2024/11/20-16:36:47.608452 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046104.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:48.715213 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046104.sst): second pass time = 2.70's, 40.165'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.11's, 35.486'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2869.824'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.272'MB/sec write SST data time = 0.26's, 151.855'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 31137.002'MB/sec zip my value time = 2.70's, unzip length = 0.108'GB zip my value throughput = 40.165'MB/sec zip pipeline throughput = 169.076'MB/sec entries = 924395 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.20 avg-zval = 37.37 usrkeys = 924395 avg-key = 20.00 avg-zkey = 1.90 avg-val = 121.19 avg-zval = 37.37 TagRS{ kinds = RS_Key0_TagN bytes = 144520 } TagArray{ size = 527752 bytes = 1781184 } seq expand size = 3694264 multi value expand size = 0 cnt WriteAppend = 15 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0345 GB dict = 1.08 MB all = 0.0393 GB } UnZip/Zip{ index = 10.5110 value = 3.1365 dict = 1.00 all = 3.4138 } Zip/UnZip{ index = 0.0951 value = 0.3188 dict = 1.00 all = 0.2929 } ---------------------------- total value len = 2184.951337 GB avg = 0.300 KB (by entry num) total key len = 191.900308 GB avg = 0.026 KB total ukey len = 134.361057 GB avg = 0.014 KB total ukey num = 9.338168321 Billion total entry num = 7.282318467 Billion write speed all = 149.944542492 MB/sec (with seq num) write speed all = 146.269281016 MB/sec (without seq num) 2024/11/20-16:36:48.770215 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46104: 924395 keys, 39316591 bytes, temperature: kWarm 2024/11/20-16:36:48.770276 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091808770247, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46104, "file_size": 39316591, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 0, "largest_seqno": 612373203, "table_properties": {"data_size": 34540352, "index_size": 1758912, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25883060, "raw_average_key_size": 28, "raw_value_size": 108334750, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924395, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_TagN-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091803, "oldest_key_time": 0, "file_creation_time": 1732091803, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46104, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:48.810989 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046105.sst) /dev/shm/ToplingTemp/Topling-589226-ifnt2J 2024/11/20-16:36:50.114947 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046105.sst): first pass time = 1.30's, 102.934'MB/sec 2024/11/20-16:36:52.715342 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046105.sst): old prealloc_size = 147639500, real_size = 33641264 2024/11/20-16:36:52.716065 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046105.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924438 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0317 GB avg-val = 117.19 avg-zval = 34.24 2024/11/20-16:36:52.716950 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046105.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:53.757878 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046105.sst): second pass time = 2.60's, 41.675'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 36.194'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2521.945'MB/sec (index lex order gen) rebuild zvType time = 0.87's, 0.264'MB/sec write SST data time = 0.17's, 225.983'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 28530.527'MB/sec zip my value time = 2.60's, unzip length = 0.108'GB zip my value throughput = 41.675'MB/sec zip pipeline throughput = 210.899'MB/sec entries = 924438 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.19 avg-zval = 34.24 usrkeys = 924438 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.19 avg-zval = 34.24 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924438 bytes = 3235552 } seq expand size = 6471066 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0317 GB dict = 1.08 MB all = 0.0377 GB } UnZip/Zip{ index = 10.5142 value = 3.4227 dict = 1.00 all = 3.5570 } Zip/UnZip{ index = 0.0951 value = 0.2922 dict = 1.00 all = 0.2811 } ---------------------------- total value len = 2185.066142 GB avg = 0.300 KB (by entry num) total key len = 191.926192 GB avg = 0.026 KB total ukey len = 134.379545 GB avg = 0.014 KB total ukey num = 9.339092759 Billion total entry num = 7.283242905 Billion write speed all = 149.905729937 MB/sec (with seq num) write speed all = 146.231170861 MB/sec (without seq num) 2024/11/20-16:36:53.807030 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46105: 924438 keys, 37733328 bytes, temperature: kWarm 2024/11/20-16:36:53.807095 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091813807065, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46105, "file_size": 37733328, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 542625158, "largest_seqno": 783301009, "table_properties": {"data_size": 31651488, "index_size": 1758464, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25884264, "raw_average_key_size": 28, "raw_value_size": 108333544, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924438, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091808, "oldest_key_time": 0, "file_creation_time": 1732091808, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46105, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:53.853729 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046106.sst) /dev/shm/ToplingTemp/Topling-589226-FLuUGJ 2024/11/20-16:36:54.908151 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046106.sst): first pass time = 1.05's, 127.295'MB/sec 2024/11/20-16:36:56.365074 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046106.sst): old prealloc_size = 147639500, real_size = 35146464 2024/11/20-16:36:56.365780 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046106.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926423 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0332 GB avg-val = 116.88 avg-zval = 35.79 2024/11/20-16:36:56.366672 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046106.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:36:57.334860 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046106.sst): second pass time = 1.46's, 74.366'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 40.347'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2603.899'MB/sec (index lex order gen) rebuild zvType time = 0.82's, 0.284'MB/sec write SST data time = 0.15's, 255.479'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30392.844'MB/sec zip my value time = 1.46's, unzip length = 0.108'GB zip my value throughput = 74.366'MB/sec zip pipeline throughput = 74.366'MB/sec entries = 926423 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.88 avg-zval = 35.79 usrkeys = 926423 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.88 avg-zval = 35.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926423 bytes = 3126688 } seq expand size = 6484961 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0332 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.5126 value = 3.2661 dict = 1.00 all = 3.4302 } Zip/UnZip{ index = 0.0951 value = 0.3062 dict = 1.00 all = 0.2915 } ---------------------------- total value len = 2185.180905 GB avg = 0.300 KB (by entry num) total key len = 191.952132 GB avg = 0.026 KB total ukey len = 134.398074 GB avg = 0.014 KB total ukey num = 9.340019182 Billion total entry num = 7.284169328 Billion write speed all = 149.880792869 MB/sec (with seq num) write speed all = 146.206595229 MB/sec (without seq num) 2024/11/20-16:36:57.384557 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46106: 926423 keys, 39127872 bytes, temperature: kWarm 2024/11/20-16:36:57.384625 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091817384593, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46106, "file_size": 39127872, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 664578890, "largest_seqno": 783301050, "table_properties": {"data_size": 33152208, "index_size": 1762496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25939844, "raw_average_key_size": 28, "raw_value_size": 108278022, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926423, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091813, "oldest_key_time": 0, "file_creation_time": 1732091813, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46106, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:36:57.422956 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046107.sst) /dev/shm/ToplingTemp/Topling-589226-8GPHMF 2024/11/20-16:36:58.807563 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046107.sst): first pass time = 1.38's, 96.940'MB/sec 2024/11/20-16:37:00.121221 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046107.sst): old prealloc_size = 147639500, real_size = 35121952 2024/11/20-16:37:00.121769 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046107.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 928013 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0331 GB avg-val = 116.63 avg-zval = 35.69 2024/11/20-16:37:00.122572 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046107.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/20-16:37:01.004470 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046107.sst): second pass time = 1.31's, 82.453'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 44.278'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3362.526'MB/sec (index lex order gen) rebuild zvType time = 0.75's, 0.309'MB/sec write SST data time = 0.13's, 298.700'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 29349.208'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.453'MB/sec zip pipeline throughput = 82.453'MB/sec entries = 928013 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.63 avg-zval = 35.69 usrkeys = 928013 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.63 avg-zval = 35.69 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 928013 bytes = 3132064 } seq expand size = 6496091 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0331 GB dict = 1.08 MB all = 0.0391 GB } UnZip/Zip{ index = 10.5123 value = 3.2675 dict = 1.00 all = 3.4320 } Zip/UnZip{ index = 0.0951 value = 0.3060 dict = 1.00 all = 0.2914 } ---------------------------- total value len = 2185.295634 GB avg = 0.300 KB (by entry num) total key len = 191.978116 GB avg = 0.026 KB total ukey len = 134.416634 GB avg = 0.014 KB total ukey num = 9.340947195 Billion total entry num = 7.285097341 Billion write speed all = 149.854992632 MB/sec (with seq num) write speed all = 146.181176916 MB/sec (without seq num) 2024/11/20-16:37:01.061901 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46107: 928013 keys, 39107520 bytes, temperature: kWarm 2024/11/20-16:37:01.061975 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091821061942, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46107, "file_size": 39107520, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 664578947, "largest_seqno": 783301091, "table_properties": {"data_size": 33124176, "index_size": 1765568, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25984364, "raw_average_key_size": 28, "raw_value_size": 108233378, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 928013, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091817, "oldest_key_time": 0, "file_creation_time": 1732091817, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46107, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:37:01.096798 140257030129216 [top_zip_table_builder.cc:960] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046108.sst) /dev/shm/ToplingTemp/Topling-589226-YMnuTF 2024/11/20-16:37:01.472298 140257030129216 [top_zip_table_builder.cc:1378] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046108.sst): first pass time = 0.38's, 112.808'MB/sec 2024/11/20-16:37:02.017827 140257030129216 [top_zip_table_builder.cc:2798] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046108.sst): old prealloc_size = 147639500, real_size = 11662512 2024/11/20-16:37:02.018125 140257030129216 [top_zip_table_builder.cc:1799] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046108.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 293382 prefix = 4 raw-key = 0.0047 GB zip-key = 0.0006 GB avg-key = 16.00 avg-zkey = 1.97 raw-val = 0.0341 GB zip-val = 0.0110 GB avg-val = 116.37 avg-zval = 37.53 2024/11/20-16:37:02.018434 140257030129216 [top_zip_table_builder.cc:1325] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046108.sst: sumWaitingMem = 0.001 GB, sumWorkingMem = 0.018 GB, reorder workingMem = 0.0007 GB, waited 0.000 sec, Key+Value = 0.042 GB 2024/11/20-16:37:02.314863 140257030129216 [top_zip_table_builder.cc:3129] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00/046108.sst): second pass time = 0.54's, 62.680'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.30's, 43.778'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2135.256'MB/sec (index lex order gen) rebuild zvType time = 0.25's, 0.298'MB/sec write SST data time = 0.05's, 256.398'MB/sec dict compress time = 140278.07's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.347'MB, throughput = 11811.947'MB/sec zip my value time = 0.54's, unzip length = 0.034'GB zip my value throughput = 62.680'MB/sec zip pipeline throughput = 107.788'MB/sec entries = 293382 avg-key = 28.00 avg-zkey = 1.97 avg-val = 116.37 avg-zval = 37.53 usrkeys = 293382 avg-key = 20.00 avg-zkey = 1.97 avg-val = 123.37 avg-zval = 37.53 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 293382 bytes = 1063520 } seq expand size = 2053674 multi value expand size = 0 cnt WriteAppend = 25 UnZipSize{ index = 0.0059 GB value = 0.0341 GB dict = 0.35 MB all = 0.0424 GB } __ZipSize{ index = 0.0006 GB value = 0.0110 GB dict = 0.35 MB all = 0.0130 GB } UnZip/Zip{ index = 10.1519 value = 3.1006 dict = 1.00 all = 3.2573 } Zip/UnZip{ index = 0.0985 value = 0.3225 dict = 1.00 all = 0.3070 } ---------------------------- total value len = 2185.331829 GB avg = 0.300 KB (by entry num) total key len = 191.986331 GB avg = 0.026 KB total ukey len = 134.422502 GB avg = 0.014 KB total ukey num = 9.341240577 Billion total entry num = 7.285390723 Billion write speed all = 149.845414381 MB/sec (with seq num) write speed all = 146.171754170 MB/sec (without seq num) 2024/11/20-16:37:02.337449 140257030129216 [compaction_job.cc:1948] [default] [JOB 16186] Generated table #46108: 293382 keys, 13003309 bytes, temperature: kWarm 2024/11/20-16:37:02.337509 140257030129216 EVENT_LOG_v1 {"time_micros": 1732091822337479, "cf_name": "default", "job": 16186, "event": "table_file_creation", "file_number": 46108, "file_size": 13003309, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 505009814, "largest_seqno": 783301108, "table_properties": {"data_size": 11010976, "index_size": 577984, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 8214696, "raw_average_key_size": 28, "raw_value_size": 34141076, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 293382, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732091821, "oldest_key_time": 0, "file_creation_time": 1732091821, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "92E45FH6K6HCJOD41QCQ", "orig_file_number": 46108, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/20-16:37:02.354765 140257030129216 [compaction_job.cc:703] job-16186: subcompact[0], size: 0.637498 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-16186/att-00 [046092,046093,046094,046095,046096,046097,046098,046099,046100,046101,046102,046103,046104,046105,046106,046107,046108] 2024/11/20-16:37:02.514890 140257030129216 (Original Log Time 2024/11/20-16:36:09.303087) EVENT_LOG_v1 {"time_micros": 1732091769303065, "job": 16186, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [46020], "files_L3": [46080, 46081, 45973, 45974, 45975, 45976, 45977, 45978, 45979, 45980, 45981, 45982, 45983, 45984, 45985, 45986, 46083], "score": 0.999783, "input_data_size": 658275157, "oldest_snapshot_seqno": 784244148} 2024/11/20-16:37:02.514895 140257030129216 (Original Log Time 2024/11/20-16:37:02.513183) [compaction_job.cc:2022] [default] [JOB 16186] Compacted 1@2 + 17@3 files to L3 => 637498087 bytes 2024/11/20-16:37:02.514896 140257030129216 (Original Log Time 2024/11/20-16:37:02.514805) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 17 0 0 0] max score 0.02, MB/sec: 12.4 rd, 12.0 wr, level 3, files in(1, 17) out(17 +0 blob) MB in(27.1, 600.7 +0.0 blob) out(608.0 +0.0 blob), read-write-amplify(45.6) write-amplify(22.4) OK, records in: 15189854, records dropped: 81343 output_compression: Snappy 2024/11/20-16:37:02.514900 140257030129216 (Original Log Time 2024/11/20-16:37:02.514867) EVENT_LOG_v1 {"time_micros": 1732091822514834, "cf": "default", "job": 16186, "event": "compaction_finished", "compaction_time_micros": 53051689, "compaction_time_cpu_micros": 22999698, "output_level": 3, "num_output_files": 17, "total_output_size": 637498087, "num_input_records": 15189854, "num_output_records": 15108511, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 460592427, "file_range_sync_nanos": 0, "file_fsync_nanos": 15462, "file_prepare_write_nanos": 44050, "lsm_state": [0, 0, 0, 17, 0, 0, 0]} 2024/11/20-16:37:02.664648 140257030129216 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732076400,"starts":1732091769,"executesMs":53344,"compactionJobId":16186,"attempt":0,"compactionInputRawBytes":2201391248,"compactionInputZipBytes":658275157,"compactionOutputRawBytes":0,"compactionOutputZipBytes":637498087,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-86aca808a5b297b6c68999a734c182cb-5558a56fc0112476-00"}}, response = 2024/11/20-16:37:02.664756 140257030129216 INFO dcompact_worker.cpp:1464: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-20T12.20.00/.rocksdb/job-16186/att-00: olev 3, work 53.344 s, result 30.565 ms, install 1.746 ms, input{raw 2.050 GiB zip 627.780 MiB}