2024/11/24-05:07:57.086766 139957643191872 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00: fork to child time = 0.021377 sec 2024/11/24-05:07:57.087767 139957643191872 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00 2024/11/24-05:07:57.094778 139957643191872 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00 2024/11/24-05:07:57.096149 139957643191872 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-05:07:57.096509 139957643191872 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-05:07:57.158386 139957643191872 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-41981/att-00/MANIFEST-206347 2024/11/24-05:07:57.183676 139957643191872 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-41981/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 293004, last_sequence is 4453051770, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 292847 2024/11/24-05:07:57.183692 139957643191872 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-05:07:57.183844 139957643191872 [version_set.cc:5721] Creating manifest 293004 2024/11/24-05:07:57.221375 139957643191872 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-05:07:57.246067 139957643191872 [compaction_job.cc:2437] [default] [JOB 41981] Compacting 1@2 + 22@3 files to L3, score 0.71, subcompactions 1 : 1 2024/11/24-05:07:57.246096 139957643191872 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 31473 Base level 2, inputs: [292998(27MB)], [293001(36MB) 293002(2050KB) 292925(33MB) 292926(35MB) 292927(33MB) 292928(33MB) 292929(35MB) 292930(33MB) 292931(34MB) 292932(35MB) 292933(33MB) 292934(35MB) 292935(4881KB) 292901(21MB) 292902(33MB) 292903(35MB) 292904(33MB) 292905(34MB) 292906(35MB) 292907(33MB) 292908(35MB) 292909(35MB)] 2024/11/24-05:07:57.281450 139957643191872 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-05:07:57.281496 139957643191872 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-05:07:57.281527 139957643191872 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-05:07:57.281537 139957643191872 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-05:07:57.281562 139957643191872 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-05:07:57.988479 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293005.sst) /dev/shm/ToplingTemp/Topling-276621-KIXXA0 2024/11/24-05:07:58.580586 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293005.sst): first pass time = 0.59's, 226.695'MB/sec 2024/11/24-05:08:00.502545 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293005.sst): old prealloc_size = 147639500, real_size = 34113792 2024/11/24-05:08:00.503137 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293005.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 930783 prefix = 4 raw-key = 0.0149 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0321 GB avg-val = 116.20 avg-zval = 34.50 2024/11/24-05:08:00.503931 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293005.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:01.476673 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293005.sst): second pass time = 1.92's, 56.313'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 39.239'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3087.804'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.276'MB/sec write SST data time = 0.13's, 293.334'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.078'MB, throughput = 27501.607'MB/sec zip my value time = 1.92's, unzip length = 0.108'GB zip my value throughput = 56.313'MB/sec zip pipeline throughput = 94.146'MB/sec entries = 930783 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.20 avg-zval = 34.50 usrkeys = 930783 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.20 avg-zval = 34.50 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 930783 bytes = 3257760 } seq expand size = 6515481 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0186 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0321 GB dict = 1.08 MB all = 0.0382 GB } UnZip/Zip{ index = 10.5151 value = 3.3682 dict = 1.00 all = 3.5114 } Zip/UnZip{ index = 0.0951 value = 0.2969 dict = 1.00 all = 0.2848 } ---------------------------- total value len = 2251.440245 GB avg = 0.200 KB (by entry num) total key len = 306.909024 GB avg = 0.027 KB total ukey len = 237.373343 GB avg = 0.015 KB total ukey num = 15.739591946 Billion total entry num = 11.280905894 Billion write speed all = 35.157004456 MB/sec (with seq num) write speed all = 33.916820831 MB/sec (without seq num) 2024/11/24-05:08:01.524369 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293005: 930783 keys, 38223058 bytes, temperature: kWarm 2024/11/24-05:08:01.524464 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396081524406, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293005, "file_size": 38223058, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178518, "largest_seqno": 4453001345, "table_properties": {"data_size": 32110576, "index_size": 1770368, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 26061924, "raw_average_key_size": 28, "raw_value_size": 108155906, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 930783, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396077, "oldest_key_time": 0, "file_creation_time": 1732396077, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293005, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:01.564580 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293006.sst) /dev/shm/ToplingTemp/Topling-276621-FlexUZ 2024/11/24-05:08:02.648439 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293006.sst): first pass time = 1.08's, 123.838'MB/sec 2024/11/24-05:08:03.981415 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293006.sst): old prealloc_size = 147639500, real_size = 31414720 2024/11/24-05:08:03.982011 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293006.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1085 GB zip-val = 0.0294 GB avg-val = 118.00 avg-zval = 31.99 2024/11/24-05:08:03.982796 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293006.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:04.970630 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293006.sst): second pass time = 1.33's, 81.452'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.99's, 35.879'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3099.707'MB/sec (index lex order gen) rebuild zvType time = 0.77's, 0.298'MB/sec write SST data time = 0.22's, 163.757'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 35958.441'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.452'MB/sec zip pipeline throughput = 81.452'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.93 avg-val = 118.00 avg-zval = 31.99 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.93 avg-val = 125.00 avg-zval = 31.99 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3217568 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0294 GB dict = 1.09 MB all = 0.0355 GB } UnZip/Zip{ index = 10.3775 value = 3.6881 dict = 1.00 all = 3.7817 } Zip/UnZip{ index = 0.0964 value = 0.2711 dict = 1.00 all = 0.2644 } ---------------------------- total value len = 2251.555157 GB avg = 0.200 KB (by entry num) total key len = 306.934765 GB avg = 0.027 KB total ukey len = 237.391729 GB avg = 0.015 KB total ukey num = 15.740511246 Billion total entry num = 11.281825194 Billion write speed all = 35.157249270 MB/sec (with seq num) write speed all = 33.917024129 MB/sec (without seq num) 2024/11/24-05:08:05.015322 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293006: 919300 keys, 35491442 bytes, temperature: kWarm 2024/11/24-05:08:05.015393 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396085015355, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293006, "file_size": 35491442, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4283584136, "largest_seqno": 4435716857, "table_properties": {"data_size": 29412976, "index_size": 1771712, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396081, "oldest_key_time": 0, "file_creation_time": 1732396081, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293006, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:05.053455 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293007.sst) /dev/shm/ToplingTemp/Topling-276621-5KbsLY 2024/11/24-05:08:06.064621 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293007.sst): first pass time = 1.01's, 132.740'MB/sec 2024/11/24-05:08:07.474716 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293007.sst): old prealloc_size = 147639500, real_size = 33277728 2024/11/24-05:08:07.475254 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293007.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927498 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1082 GB zip-val = 0.0313 GB avg-val = 116.71 avg-zval = 33.73 2024/11/24-05:08:07.476032 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293007.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:08.441835 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293007.sst): second pass time = 1.41's, 76.811'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 38.651'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3427.841'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.276'MB/sec write SST data time = 0.13's, 292.163'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34736.060'MB/sec zip my value time = 1.41's, unzip length = 0.108'GB zip my value throughput = 76.811'MB/sec zip pipeline throughput = 76.811'MB/sec entries = 927498 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.71 avg-zval = 33.73 usrkeys = 927498 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.71 avg-zval = 33.73 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927498 bytes = 3246256 } seq expand size = 6492486 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5122 value = 3.4605 dict = 1.00 all = 3.5908 } Zip/UnZip{ index = 0.0951 value = 0.2890 dict = 1.00 all = 0.2785 } ---------------------------- total value len = 2251.669897 GB avg = 0.200 KB (by entry num) total key len = 306.960735 GB avg = 0.027 KB total ukey len = 237.410279 GB avg = 0.015 KB total ukey num = 15.741438744 Billion total entry num = 11.282752692 Billion write speed all = 35.157505848 MB/sec (with seq num) write speed all = 33.917237905 MB/sec (without seq num) 2024/11/24-05:08:08.487817 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293007: 927498 keys, 37378094 bytes, temperature: kWarm 2024/11/24-05:08:08.487886 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396088487850, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293007, "file_size": 37378094, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178530, "largest_seqno": 4453001101, "table_properties": {"data_size": 31281040, "index_size": 1764608, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25969944, "raw_average_key_size": 28, "raw_value_size": 108247832, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927498, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396085, "oldest_key_time": 0, "file_creation_time": 1732396085, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293007, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:08.527768 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293008.sst) /dev/shm/ToplingTemp/Topling-276621-7RAJKY 2024/11/24-05:08:09.338712 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293008.sst): first pass time = 0.81's, 165.516'MB/sec 2024/11/24-05:08:10.692377 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293008.sst): old prealloc_size = 147639500, real_size = 30710912 2024/11/24-05:08:10.692914 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293008.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927303 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0287 GB avg-val = 116.74 avg-zval = 30.95 2024/11/24-05:08:10.693712 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293008.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:11.657561 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293008.sst): second pass time = 1.35's, 80.023'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 35.946'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3477.249'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.290'MB/sec write SST data time = 0.16's, 211.595'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35259.740'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.023'MB/sec zip pipeline throughput = 80.023'MB/sec entries = 927303 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.74 avg-zval = 30.95 usrkeys = 927303 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.74 avg-zval = 30.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927303 bytes = 3129664 } seq expand size = 6491121 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0287 GB dict = 1.08 MB all = 0.0347 GB } UnZip/Zip{ index = 10.4441 value = 3.7715 dict = 1.00 all = 3.8687 } Zip/UnZip{ index = 0.0957 value = 0.2651 dict = 1.00 all = 0.2585 } ---------------------------- total value len = 2251.784642 GB avg = 0.200 KB (by entry num) total key len = 306.986699 GB avg = 0.027 KB total ukey len = 237.428825 GB avg = 0.015 KB total ukey num = 15.742366047 Billion total entry num = 11.283679995 Billion write speed all = 35.157885786 MB/sec (with seq num) write speed all = 33.917570714 MB/sec (without seq num) 2024/11/24-05:08:11.700259 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293008: 927303 keys, 34693322 bytes, temperature: kWarm 2024/11/24-05:08:11.700329 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396091700294, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293008, "file_size": 34693322, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4348476715, "largest_seqno": 4453000278, "table_properties": {"data_size": 28703152, "index_size": 1775744, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25964484, "raw_average_key_size": 28, "raw_value_size": 108253366, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927303, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396088, "oldest_key_time": 0, "file_creation_time": 1732396088, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293008, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:11.741265 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293009.sst) /dev/shm/ToplingTemp/Topling-276621-sUa1I0 2024/11/24-05:08:12.594951 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293009.sst): first pass time = 0.85's, 157.228'MB/sec 2024/11/24-05:08:13.913128 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293009.sst): old prealloc_size = 147639500, real_size = 31383856 2024/11/24-05:08:13.913708 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293009.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919422 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0017 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1085 GB zip-val = 0.0294 GB avg-val = 117.98 avg-zval = 31.98 2024/11/24-05:08:13.914494 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293009.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:14.874613 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293009.sst): second pass time = 1.32's, 82.367'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.96's, 36.885'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3126.779'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.313'MB/sec write SST data time = 0.23's, 157.018'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 33865.310'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.367'MB/sec zip pipeline throughput = 82.367'MB/sec entries = 919422 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.98 avg-zval = 31.98 usrkeys = 919422 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.98 avg-zval = 31.98 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919422 bytes = 3217984 } seq expand size = 6435954 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0017 GB value = 0.0294 GB dict = 1.09 MB all = 0.0355 GB } UnZip/Zip{ index = 10.5145 value = 3.6890 dict = 1.00 all = 3.7847 } Zip/UnZip{ index = 0.0951 value = 0.2711 dict = 1.00 all = 0.2642 } ---------------------------- total value len = 2251.899552 GB avg = 0.200 KB (by entry num) total key len = 307.012443 GB avg = 0.027 KB total ukey len = 237.447213 GB avg = 0.015 KB total ukey num = 15.743285469 Billion total entry num = 11.284599417 Billion write speed all = 35.158264293 MB/sec (with seq num) write speed all = 33.917902985 MB/sec (without seq num) 2024/11/24-05:08:14.918757 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293009: 919422 keys, 35462988 bytes, temperature: kWarm 2024/11/24-05:08:14.918877 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396094918792, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293009, "file_size": 35462988, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178548, "largest_seqno": 4453001534, "table_properties": {"data_size": 29404960, "index_size": 1748864, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25743816, "raw_average_key_size": 28, "raw_value_size": 108474008, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 919422, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396091, "oldest_key_time": 0, "file_creation_time": 1732396091, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293009, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:14.963640 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293010.sst) /dev/shm/ToplingTemp/Topling-276621-50CB3X 2024/11/24-05:08:15.984543 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293010.sst): first pass time = 1.02's, 131.475'MB/sec 2024/11/24-05:08:17.364233 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293010.sst): old prealloc_size = 147639500, real_size = 33338848 2024/11/24-05:08:17.364880 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293010.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927347 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 116.73 avg-zval = 33.79 2024/11/24-05:08:17.365741 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293010.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:18.361275 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293010.sst): second pass time = 1.38's, 78.511'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.00's, 37.551'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2892.883'MB/sec (index lex order gen) rebuild zvType time = 0.85's, 0.272'MB/sec write SST data time = 0.14's, 259.365'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35071.505'MB/sec zip my value time = 1.38's, unzip length = 0.108'GB zip my value throughput = 78.511'MB/sec zip pipeline throughput = 78.511'MB/sec entries = 927347 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.73 avg-zval = 33.79 usrkeys = 927347 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.73 avg-zval = 33.79 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927347 bytes = 3245728 } seq expand size = 6491429 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.4442 value = 3.4551 dict = 1.00 all = 3.5850 } Zip/UnZip{ index = 0.0957 value = 0.2894 dict = 1.00 all = 0.2789 } ---------------------------- total value len = 2252.014295 GB avg = 0.200 KB (by entry num) total key len = 307.038409 GB avg = 0.027 KB total ukey len = 237.465760 GB avg = 0.015 KB total ukey num = 15.744212816 Billion total entry num = 11.285526764 Billion write speed all = 35.158513303 MB/sec (with seq num) write speed all = 33.918109485 MB/sec (without seq num) 2024/11/24-05:08:18.406437 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293010: 927347 keys, 37439126 bytes, temperature: kWarm 2024/11/24-05:08:18.406507 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396098406470, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293010, "file_size": 37439126, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178559, "largest_seqno": 4453000628, "table_properties": {"data_size": 31331024, "index_size": 1775808, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25965716, "raw_average_key_size": 28, "raw_value_size": 108252042, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927347, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396094, "oldest_key_time": 0, "file_creation_time": 1732396094, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293010, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:18.449212 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293011.sst) /dev/shm/ToplingTemp/Topling-276621-EpxVXZ 2024/11/24-05:08:19.427522 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293011.sst): first pass time = 0.98's, 137.202'MB/sec 2024/11/24-05:08:20.770093 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293011.sst): old prealloc_size = 147639500, real_size = 31070912 2024/11/24-05:08:20.770648 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293011.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923533 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0291 GB avg-val = 117.33 avg-zval = 31.48 2024/11/24-05:08:20.771406 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293011.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:21.697279 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293011.sst): second pass time = 1.34's, 80.755'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.93's, 37.797'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3322.194'MB/sec (index lex order gen) rebuild zvType time = 0.80's, 0.289'MB/sec write SST data time = 0.13's, 274.707'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 35283.107'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.755'MB/sec zip pipeline throughput = 80.755'MB/sec entries = 923533 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.33 avg-zval = 31.48 usrkeys = 923533 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.33 avg-zval = 31.48 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923533 bytes = 3116944 } seq expand size = 6464731 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0291 GB dict = 1.08 MB all = 0.0350 GB } UnZip/Zip{ index = 10.4464 value = 3.7273 dict = 1.00 all = 3.8300 } Zip/UnZip{ index = 0.0957 value = 0.2683 dict = 1.00 all = 0.2611 } ---------------------------- total value len = 2252.129119 GB avg = 0.200 KB (by entry num) total key len = 307.064268 GB avg = 0.027 KB total ukey len = 237.484231 GB avg = 0.015 KB total ukey num = 15.745136349 Billion total entry num = 11.286450297 Billion write speed all = 35.158834691 MB/sec (with seq num) write speed all = 33.918386220 MB/sec (without seq num) 2024/11/24-05:08:21.741904 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293011: 923533 keys, 35043744 bytes, temperature: kWarm 2024/11/24-05:08:21.741973 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396101741939, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293011, "file_size": 35043744, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4326519240, "largest_seqno": 4453000311, "table_properties": {"data_size": 29071728, "index_size": 1768128, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25858924, "raw_average_key_size": 28, "raw_value_size": 108358830, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923533, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396098, "oldest_key_time": 0, "file_creation_time": 1732396098, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293011, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:21.786904 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293012.sst) /dev/shm/ToplingTemp/Topling-276621-ZAii40 2024/11/24-05:08:22.683410 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293012.sst): first pass time = 0.90's, 149.718'MB/sec 2024/11/24-05:08:24.020822 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293012.sst): old prealloc_size = 147639500, real_size = 32681408 2024/11/24-05:08:24.021400 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293012.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923228 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0307 GB avg-val = 117.38 avg-zval = 33.25 2024/11/24-05:08:24.022183 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293012.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:24.987357 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293012.sst): second pass time = 1.34's, 81.080'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.97's, 38.044'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3148.895'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.276'MB/sec write SST data time = 0.13's, 284.604'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 35485.282'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 81.080'MB/sec zip pipeline throughput = 81.080'MB/sec entries = 923228 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.38 avg-zval = 33.25 usrkeys = 923228 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.38 avg-zval = 33.25 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923228 bytes = 3231312 } seq expand size = 6462596 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.5138 value = 3.5306 dict = 1.00 all = 3.6502 } Zip/UnZip{ index = 0.0951 value = 0.2832 dict = 1.00 all = 0.2740 } ---------------------------- total value len = 2252.243949 GB avg = 0.200 KB (by entry num) total key len = 307.090118 GB avg = 0.027 KB total ukey len = 237.502696 GB avg = 0.015 KB total ukey num = 15.746059577 Billion total entry num = 11.287373525 Billion write speed all = 35.159178198 MB/sec (with seq num) write speed all = 33.918684329 MB/sec (without seq num) 2024/11/24-05:08:25.032184 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293012: 923228 keys, 36770274 bytes, temperature: kWarm 2024/11/24-05:08:25.032267 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396105032218, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293012, "file_size": 36770274, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178575, "largest_seqno": 4453001514, "table_properties": {"data_size": 30694192, "index_size": 1756224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25850384, "raw_average_key_size": 28, "raw_value_size": 108367384, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923228, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396101, "oldest_key_time": 0, "file_creation_time": 1732396101, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293012, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:25.076400 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293013.sst) /dev/shm/ToplingTemp/Topling-276621-cuRD6Z 2024/11/24-05:08:25.986383 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293013.sst): first pass time = 0.91's, 147.502'MB/sec 2024/11/24-05:08:27.333844 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293013.sst): old prealloc_size = 147639500, real_size = 32969824 2024/11/24-05:08:27.334416 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293013.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927300 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0310 GB avg-val = 116.74 avg-zval = 33.40 2024/11/24-05:08:27.335916 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293013.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:28.186786 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293013.sst): second pass time = 1.35's, 80.387'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.463'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3224.194'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 290.510'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34911.494'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.387'MB/sec zip pipeline throughput = 80.387'MB/sec entries = 927300 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.74 avg-zval = 33.40 usrkeys = 927300 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.74 avg-zval = 33.40 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927300 bytes = 3245568 } seq expand size = 6491100 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0310 GB dict = 1.08 MB all = 0.0371 GB } UnZip/Zip{ index = 10.5123 value = 3.4950 dict = 1.00 all = 3.6207 } Zip/UnZip{ index = 0.0951 value = 0.2861 dict = 1.00 all = 0.2762 } ---------------------------- total value len = 2252.358693 GB avg = 0.200 KB (by entry num) total key len = 307.116082 GB avg = 0.027 KB total ukey len = 237.521242 GB avg = 0.015 KB total ukey num = 15.746986877 Billion total entry num = 11.288300825 Billion write speed all = 35.159565852 MB/sec (with seq num) write speed all = 33.919024597 MB/sec (without seq num) 2024/11/24-05:08:28.231820 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293013: 927300 keys, 37069798 bytes, temperature: kWarm 2024/11/24-05:08:28.231886 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396108231850, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293013, "file_size": 37069798, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178589, "largest_seqno": 4453000991, "table_properties": {"data_size": 30973584, "index_size": 1764224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25964400, "raw_average_key_size": 28, "raw_value_size": 108253428, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396105, "oldest_key_time": 0, "file_creation_time": 1732396105, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293013, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:28.274972 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293014.sst) /dev/shm/ToplingTemp/Topling-276621-EkqnGW 2024/11/24-05:08:29.097089 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293014.sst): first pass time = 0.82's, 163.270'MB/sec 2024/11/24-05:08:30.415580 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293014.sst): old prealloc_size = 147639500, real_size = 31331360 2024/11/24-05:08:30.416161 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293014.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919845 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0293 GB avg-val = 117.91 avg-zval = 31.90 2024/11/24-05:08:30.416935 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293014.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:31.231831 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293014.sst): second pass time = 1.32's, 82.345'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.382'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3158.098'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.334'MB/sec write SST data time = 0.13's, 279.205'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 27492.346'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.345'MB/sec zip pipeline throughput = 82.345'MB/sec entries = 919845 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.91 avg-zval = 31.90 usrkeys = 919845 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.91 avg-zval = 31.90 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919845 bytes = 3219472 } seq expand size = 6438915 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0293 GB dict = 1.08 MB all = 0.0354 GB } UnZip/Zip{ index = 10.4448 value = 3.6968 dict = 1.00 all = 3.7904 } Zip/UnZip{ index = 0.0957 value = 0.2705 dict = 1.00 all = 0.2638 } ---------------------------- total value len = 2252.473594 GB avg = 0.200 KB (by entry num) total key len = 307.141838 GB avg = 0.027 KB total ukey len = 237.539639 GB avg = 0.015 KB total ukey num = 15.747906722 Billion total entry num = 11.289220670 Billion write speed all = 35.160027319 MB/sec (with seq num) write speed all = 33.919436871 MB/sec (without seq num) 2024/11/24-05:08:31.275084 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293014: 919845 keys, 35409554 bytes, temperature: kWarm 2024/11/24-05:08:31.275155 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396111275118, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293014, "file_size": 35409554, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4304677067, "largest_seqno": 4453000339, "table_properties": {"data_size": 29339856, "index_size": 1761344, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25755660, "raw_average_key_size": 28, "raw_value_size": 108462190, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 919845, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396108, "oldest_key_time": 0, "file_creation_time": 1732396108, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293014, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:31.316755 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293015.sst) /dev/shm/ToplingTemp/Topling-276621-2FnQJ0 2024/11/24-05:08:32.224027 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293015.sst): first pass time = 0.91's, 147.943'MB/sec 2024/11/24-05:08:33.564196 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293015.sst): old prealloc_size = 147639500, real_size = 33146560 2024/11/24-05:08:33.564742 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293015.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926889 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 116.80 avg-zval = 33.60 2024/11/24-05:08:33.565538 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293015.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:34.418755 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293015.sst): second pass time = 1.34's, 80.856'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.585'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3380.115'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.319'MB/sec write SST data time = 0.13's, 291.577'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 27829.202'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.856'MB/sec zip pipeline throughput = 80.856'MB/sec entries = 926889 avg-key = 28.00 avg-zkey = 1.91 avg-val = 116.80 avg-zval = 33.60 usrkeys = 926889 avg-key = 20.00 avg-zkey = 1.91 avg-val = 123.80 avg-zval = 33.60 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926889 bytes = 3244128 } seq expand size = 6488223 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.4440 value = 3.4767 dict = 1.00 all = 3.6037 } Zip/UnZip{ index = 0.0957 value = 0.2876 dict = 1.00 all = 0.2775 } ---------------------------- total value len = 2252.588348 GB avg = 0.200 KB (by entry num) total key len = 307.167791 GB avg = 0.027 KB total ukey len = 237.558176 GB avg = 0.015 KB total ukey num = 15.748833611 Billion total entry num = 11.290147559 Billion write speed all = 35.160420901 MB/sec (with seq num) write speed all = 33.919782907 MB/sec (without seq num) 2024/11/24-05:08:34.464179 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293015: 926889 keys, 37244756 bytes, temperature: kWarm 2024/11/24-05:08:34.464262 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396114464217, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293015, "file_size": 37244756, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178599, "largest_seqno": 4453000563, "table_properties": {"data_size": 31139696, "index_size": 1774976, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25952892, "raw_average_key_size": 28, "raw_value_size": 108264862, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926889, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396111, "oldest_key_time": 0, "file_creation_time": 1732396111, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293015, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:34.508364 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293016.sst) /dev/shm/ToplingTemp/Topling-276621-BJE6FW 2024/11/24-05:08:35.771964 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293016.sst): first pass time = 1.26's, 106.222'MB/sec 2024/11/24-05:08:37.101839 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293016.sst): old prealloc_size = 147639500, real_size = 32279008 2024/11/24-05:08:37.102416 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293016.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927455 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1082 GB zip-val = 0.0303 GB avg-val = 116.72 avg-zval = 32.63 2024/11/24-05:08:37.103771 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293016.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:37.936308 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293016.sst): second pass time = 1.33's, 81.447'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 43.603'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3216.226'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.328'MB/sec write SST data time = 0.13's, 285.755'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34746.945'MB/sec zip my value time = 1.33's, unzip length = 0.108'GB zip my value throughput = 81.447'MB/sec zip pipeline throughput = 81.447'MB/sec entries = 927455 avg-key = 28.00 avg-zkey = 1.93 avg-val = 116.72 avg-zval = 32.63 usrkeys = 927455 avg-key = 20.00 avg-zkey = 1.93 avg-val = 123.72 avg-zval = 32.63 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927455 bytes = 3246112 } seq expand size = 6492185 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1082 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0303 GB dict = 1.08 MB all = 0.0364 GB } UnZip/Zip{ index = 10.3774 value = 3.5774 dict = 1.00 all = 3.6889 } Zip/UnZip{ index = 0.0964 value = 0.2795 dict = 1.00 all = 0.2711 } ---------------------------- total value len = 2252.703089 GB avg = 0.200 KB (by entry num) total key len = 307.193760 GB avg = 0.027 KB total ukey len = 237.576725 GB avg = 0.015 KB total ukey num = 15.749761066 Billion total entry num = 11.291075014 Billion write speed all = 35.160654833 MB/sec (with seq num) write speed all = 33.919974869 MB/sec (without seq num) 2024/11/24-05:08:37.980462 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293016: 927455 keys, 36384112 bytes, temperature: kWarm 2024/11/24-05:08:37.980535 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396117980498, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293016, "file_size": 36384112, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178645, "largest_seqno": 4453001391, "table_properties": {"data_size": 30259536, "index_size": 1787456, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25968740, "raw_average_key_size": 28, "raw_value_size": 108249066, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927455, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396114, "oldest_key_time": 0, "file_creation_time": 1732396114, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293016, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:38.019520 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293017.sst) /dev/shm/ToplingTemp/Topling-276621-KZ1qVW 2024/11/24-05:08:39.034374 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293017.sst): first pass time = 1.01's, 132.258'MB/sec 2024/11/24-05:08:40.346019 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293017.sst): old prealloc_size = 147639500, real_size = 31469616 2024/11/24-05:08:40.346598 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293017.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1085 GB zip-val = 0.0295 GB avg-val = 118.00 avg-zval = 32.07 2024/11/24-05:08:40.347366 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293017.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:41.157704 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293017.sst): second pass time = 1.31's, 82.756'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.794'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3161.866'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.335'MB/sec write SST data time = 0.13's, 283.204'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 34431.645'MB/sec zip my value time = 1.31's, unzip length = 0.108'GB zip my value throughput = 82.756'MB/sec zip pipeline throughput = 82.756'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.91 avg-val = 118.00 avg-zval = 32.07 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.91 avg-val = 125.00 avg-zval = 32.07 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3217568 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.09 MB all = 0.0355 GB } UnZip/Zip{ index = 10.4455 value = 3.6798 dict = 1.00 all = 3.7759 } Zip/UnZip{ index = 0.0957 value = 0.2718 dict = 1.00 all = 0.2648 } ---------------------------- total value len = 2252.818001 GB avg = 0.200 KB (by entry num) total key len = 307.219500 GB avg = 0.027 KB total ukey len = 237.595111 GB avg = 0.015 KB total ukey num = 15.750680366 Billion total entry num = 11.291994314 Billion write speed all = 35.161030972 MB/sec (with seq num) write speed all = 33.920304891 MB/sec (without seq num) 2024/11/24-05:08:41.201314 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293017: 919300 keys, 35546354 bytes, temperature: kWarm 2024/11/24-05:08:41.201386 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396121201349, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293017, "file_size": 35546354, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4282135962, "largest_seqno": 4434254165, "table_properties": {"data_size": 29479392, "index_size": 1760192, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396118, "oldest_key_time": 0, "file_creation_time": 1732396118, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293017, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:41.242572 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293018.sst) /dev/shm/ToplingTemp/Topling-276621-aapBvX 2024/11/24-05:08:42.158095 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293018.sst): first pass time = 0.92's, 146.609'MB/sec 2024/11/24-05:08:43.511863 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293018.sst): old prealloc_size = 147639500, real_size = 33403872 2024/11/24-05:08:43.512476 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293018.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927404 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.72 avg-zval = 33.87 2024/11/24-05:08:43.513918 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293018.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:44.371214 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293018.sst): second pass time = 1.35's, 80.007'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.644'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2993.890'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.317'MB/sec write SST data time = 0.13's, 293.111'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 33982.885'MB/sec zip my value time = 1.35's, unzip length = 0.108'GB zip my value throughput = 80.007'MB/sec zip pipeline throughput = 80.007'MB/sec entries = 927404 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.72 avg-zval = 33.87 usrkeys = 927404 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.72 avg-zval = 33.87 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927404 bytes = 3245936 } seq expand size = 6491828 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5123 value = 3.4467 dict = 1.00 all = 3.5788 } Zip/UnZip{ index = 0.0951 value = 0.2901 dict = 1.00 all = 0.2794 } ---------------------------- total value len = 2252.932744 GB avg = 0.199 KB (by entry num) total key len = 307.245467 GB avg = 0.027 KB total ukey len = 237.613659 GB avg = 0.015 KB total ukey num = 15.751607770 Billion total entry num = 11.292921718 Billion write speed all = 35.161411668 MB/sec (with seq num) write speed all = 33.920638451 MB/sec (without seq num) 2024/11/24-05:08:44.417602 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293018: 927404 keys, 37503906 bytes, temperature: kWarm 2024/11/24-05:08:44.417677 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396124417639, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293018, "file_size": 37503906, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178655, "largest_seqno": 4453000103, "table_properties": {"data_size": 31407440, "index_size": 1764416, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25967312, "raw_average_key_size": 28, "raw_value_size": 108250472, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927404, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396121, "oldest_key_time": 0, "file_creation_time": 1732396121, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293018, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:44.458465 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293019.sst) /dev/shm/ToplingTemp/Topling-276621-GzTwxZ 2024/11/24-05:08:45.464215 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293019.sst): first pass time = 1.01's, 133.456'MB/sec 2024/11/24-05:08:46.765404 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293019.sst): old prealloc_size = 147639500, real_size = 30684336 2024/11/24-05:08:46.765955 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293019.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927085 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0287 GB avg-val = 116.77 avg-zval = 30.95 2024/11/24-05:08:46.767408 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293019.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:47.579879 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293019.sst): second pass time = 1.30's, 83.248'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 42.708'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3350.846'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.338'MB/sec write SST data time = 0.13's, 270.819'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32590.439'MB/sec zip my value time = 1.30's, unzip length = 0.108'GB zip my value throughput = 83.248'MB/sec zip pipeline throughput = 83.248'MB/sec entries = 927085 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.77 avg-zval = 30.95 usrkeys = 927085 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.77 avg-zval = 30.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927085 bytes = 3244816 } seq expand size = 6489595 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0287 GB dict = 1.08 MB all = 0.0348 GB } UnZip/Zip{ index = 10.5136 value = 3.7736 dict = 1.00 all = 3.8586 } Zip/UnZip{ index = 0.0951 value = 0.2650 dict = 1.00 all = 0.2592 } ---------------------------- total value len = 2253.047493 GB avg = 0.199 KB (by entry num) total key len = 307.271426 GB avg = 0.027 KB total ukey len = 237.632201 GB avg = 0.015 KB total ukey num = 15.752534855 Billion total entry num = 11.293848803 Billion write speed all = 35.161794640 MB/sec (with seq num) write speed all = 33.920974242 MB/sec (without seq num) 2024/11/24-05:08:47.624854 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293019: 927085 keys, 34783784 bytes, temperature: kWarm 2024/11/24-05:08:47.624925 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396127624889, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293019, "file_size": 34783784, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178675, "largest_seqno": 4453001421, "table_properties": {"data_size": 28688800, "index_size": 1763584, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25958380, "raw_average_key_size": 28, "raw_value_size": 108259422, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927085, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396124, "oldest_key_time": 0, "file_creation_time": 1732396124, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293019, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:47.665552 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293020.sst) /dev/shm/ToplingTemp/Topling-276621-tAnoaX 2024/11/24-05:08:48.562570 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293020.sst): first pass time = 0.90's, 149.634'MB/sec 2024/11/24-05:08:49.878892 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293020.sst): old prealloc_size = 147639500, real_size = 31506768 2024/11/24-05:08:49.879500 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293020.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 919300 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1085 GB zip-val = 0.0295 GB avg-val = 118.00 avg-zval = 32.11 2024/11/24-05:08:49.880624 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293020.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:50.699787 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293020.sst): second pass time = 1.32's, 82.458'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.82's, 43.349'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3010.672'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.332'MB/sec write SST data time = 0.13's, 279.361'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.085'MB, throughput = 33542.832'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.458'MB/sec zip pipeline throughput = 82.458'MB/sec entries = 919300 avg-key = 28.00 avg-zkey = 1.92 avg-val = 118.00 avg-zval = 32.11 usrkeys = 919300 avg-key = 20.00 avg-zkey = 1.92 avg-val = 125.00 avg-zval = 32.11 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 919300 bytes = 3217568 } seq expand size = 6435100 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1085 GB dict = 1.09 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.09 MB all = 0.0356 GB } UnZip/Zip{ index = 10.4424 value = 3.6752 dict = 1.00 all = 3.7719 } Zip/UnZip{ index = 0.0958 value = 0.2721 dict = 1.00 all = 0.2651 } ---------------------------- total value len = 2253.162405 GB avg = 0.199 KB (by entry num) total key len = 307.297166 GB avg = 0.027 KB total ukey len = 237.650587 GB avg = 0.015 KB total ukey num = 15.753454155 Billion total entry num = 11.294768103 Billion write speed all = 35.162219686 MB/sec (with seq num) write speed all = 33.921351456 MB/sec (without seq num) 2024/11/24-05:08:50.743137 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293020: 919300 keys, 35583474 bytes, temperature: kWarm 2024/11/24-05:08:50.743208 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396130743172, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293020, "file_size": 35583474, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261675202, "largest_seqno": 4415179017, "table_properties": {"data_size": 29516032, "index_size": 1760704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25740400, "raw_average_key_size": 28, "raw_value_size": 108477400, "raw_average_value_size": 118, "num_data_blocks": 1, "num_entries": 919300, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 118, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396127, "oldest_key_time": 0, "file_creation_time": 1732396127, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293020, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:50.794360 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293021.sst) /dev/shm/ToplingTemp/Topling-276621-quc2HW 2024/11/24-05:08:51.740793 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293021.sst): first pass time = 0.95's, 141.821'MB/sec 2024/11/24-05:08:53.080441 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293021.sst): old prealloc_size = 147639500, real_size = 33257424 2024/11/24-05:08:53.081025 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293021.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 927228 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 116.75 avg-zval = 33.71 2024/11/24-05:08:53.083030 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293021.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:53.939055 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293021.sst): second pass time = 1.34's, 80.859'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.510'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3150.899'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.318'MB/sec write SST data time = 0.13's, 290.363'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 32512.802'MB/sec zip my value time = 1.34's, unzip length = 0.108'GB zip my value throughput = 80.859'MB/sec zip pipeline throughput = 80.859'MB/sec entries = 927228 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.75 avg-zval = 33.71 usrkeys = 927228 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.75 avg-zval = 33.71 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 927228 bytes = 3245312 } seq expand size = 6490596 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5130 value = 3.4629 dict = 1.00 all = 3.5928 } Zip/UnZip{ index = 0.0951 value = 0.2888 dict = 1.00 all = 0.2783 } ---------------------------- total value len = 2253.277151 GB avg = 0.199 KB (by entry num) total key len = 307.323129 GB avg = 0.027 KB total ukey len = 237.669132 GB avg = 0.015 KB total ukey num = 15.754381383 Billion total entry num = 11.295695331 Billion write speed all = 35.162587824 MB/sec (with seq num) write speed all = 33.921672927 MB/sec (without seq num) 2024/11/24-05:08:53.983676 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293021: 927228 keys, 37357380 bytes, temperature: kWarm 2024/11/24-05:08:53.983746 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396133983712, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293021, "file_size": 37357380, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178686, "largest_seqno": 4453000956, "table_properties": {"data_size": 31261440, "index_size": 1763968, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25962384, "raw_average_key_size": 28, "raw_value_size": 108255408, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 927228, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396130, "oldest_key_time": 0, "file_creation_time": 1732396130, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293021, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:54.025298 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293022.sst) /dev/shm/ToplingTemp/Topling-276621-L7E6dZ 2024/11/24-05:08:55.052523 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293022.sst): first pass time = 1.03's, 130.666'MB/sec 2024/11/24-05:08:56.373297 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293022.sst): old prealloc_size = 147639500, real_size = 30977312 2024/11/24-05:08:56.373887 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293022.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924058 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0290 GB avg-val = 117.25 avg-zval = 31.36 2024/11/24-05:08:56.375873 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293022.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:08:57.185323 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293022.sst): second pass time = 1.32's, 82.077'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.043'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3130.639'MB/sec (index lex order gen) rebuild zvType time = 0.68's, 0.339'MB/sec write SST data time = 0.13's, 270.961'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 35596.000'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.077'MB/sec zip pipeline throughput = 82.077'MB/sec entries = 924058 avg-key = 28.00 avg-zkey = 1.92 avg-val = 117.25 avg-zval = 31.36 usrkeys = 924058 avg-key = 20.00 avg-zkey = 1.92 avg-val = 124.25 avg-zval = 31.36 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924058 bytes = 3118704 } seq expand size = 6468406 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0290 GB dict = 1.08 MB all = 0.0350 GB } UnZip/Zip{ index = 10.4437 value = 3.7390 dict = 1.00 all = 3.8402 } Zip/UnZip{ index = 0.0958 value = 0.2674 dict = 1.00 all = 0.2604 } ---------------------------- total value len = 2253.391964 GB avg = 0.199 KB (by entry num) total key len = 307.349002 GB avg = 0.027 KB total ukey len = 237.687613 GB avg = 0.015 KB total ukey num = 15.755305441 Billion total entry num = 11.296619389 Billion write speed all = 35.162952247 MB/sec (with seq num) write speed all = 33.921991156 MB/sec (without seq num) 2024/11/24-05:08:57.238417 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293022: 924058 keys, 34951098 bytes, temperature: kWarm 2024/11/24-05:08:57.238487 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396137238452, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293022, "file_size": 34951098, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4329580553, "largest_seqno": 4452999131, "table_properties": {"data_size": 28976528, "index_size": 1769600, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25873624, "raw_average_key_size": 28, "raw_value_size": 108344232, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924058, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396134, "oldest_key_time": 0, "file_creation_time": 1732396134, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293022, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:08:57.289470 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293023.sst) /dev/shm/ToplingTemp/Topling-276621-pBQpp0 2024/11/24-05:08:58.298283 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293023.sst): first pass time = 1.01's, 133.050'MB/sec 2024/11/24-05:08:59.619839 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293023.sst): old prealloc_size = 147639500, real_size = 32563792 2024/11/24-05:08:59.620437 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293023.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922567 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0306 GB avg-val = 117.48 avg-zval = 33.14 2024/11/24-05:08:59.621207 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293023.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-05:09:00.455103 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293023.sst): second pass time = 1.32's, 82.061'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.84's, 43.881'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3058.177'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.13's, 286.169'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 36584.690'MB/sec zip my value time = 1.32's, unzip length = 0.108'GB zip my value throughput = 82.061'MB/sec zip pipeline throughput = 82.061'MB/sec entries = 922567 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.48 avg-zval = 33.14 usrkeys = 922567 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.48 avg-zval = 33.14 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922567 bytes = 3228992 } seq expand size = 6457969 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0306 GB dict = 1.08 MB all = 0.0367 GB } UnZip/Zip{ index = 10.5143 value = 3.5446 dict = 1.00 all = 3.6621 } Zip/UnZip{ index = 0.0951 value = 0.2821 dict = 1.00 all = 0.2731 } ---------------------------- total value len = 2253.506808 GB avg = 0.199 KB (by entry num) total key len = 307.374834 GB avg = 0.027 KB total ukey len = 237.706064 GB avg = 0.015 KB total ukey num = 15.756228008 Billion total entry num = 11.297541956 Billion write speed all = 35.163305143 MB/sec (with seq num) write speed all = 33.922298425 MB/sec (without seq num) 2024/11/24-05:09:00.506995 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293023: 922567 keys, 36650996 bytes, temperature: kWarm 2024/11/24-05:09:00.507069 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396140507032, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293023, "file_size": 36650996, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178700, "largest_seqno": 4453001155, "table_properties": {"data_size": 30578112, "index_size": 1754880, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25831876, "raw_average_key_size": 28, "raw_value_size": 108385986, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922567, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396137, "oldest_key_time": 0, "file_creation_time": 1732396137, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293023, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:09:00.553237 139957643191872 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293024.sst) /dev/shm/ToplingTemp/Topling-276621-kVbj60 2024/11/24-05:09:01.050849 139957643191872 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293024.sst): first pass time = 0.50's, 247.197'MB/sec 2024/11/24-05:09:02.298570 139957643191872 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293024.sst): old prealloc_size = 147639500, real_size = 30410960 2024/11/24-05:09:02.299108 139957643191872 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293024.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 850290 prefix = 4 raw-key = 0.0136 GB zip-key = 0.0016 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.0992 GB zip-val = 0.0286 GB avg-val = 116.65 avg-zval = 33.61 2024/11/24-05:09:02.299914 139957643191872 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293024.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0021 GB, waited 0.000 sec, Key+Value = 0.123 GB 2024/11/24-05:09:03.076999 139957643191872 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00/293024.sst): second pass time = 1.25's, 79.545'MB/sec, value only(80.6% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.78's, 43.904'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3132.149'MB/sec (index lex order gen) rebuild zvType time = 0.66's, 0.323'MB/sec write SST data time = 0.12's, 287.000'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.994'MB, throughput = 21696.669'MB/sec zip my value time = 1.25's, unzip length = 0.099'GB zip my value throughput = 79.545'MB/sec zip pipeline throughput = 79.545'MB/sec entries = 850290 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.65 avg-zval = 33.61 usrkeys = 850290 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.65 avg-zval = 33.61 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 850290 bytes = 2976032 } seq expand size = 5952030 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0170 GB value = 0.0992 GB dict = 0.99 MB all = 0.1230 GB } __ZipSize{ index = 0.0016 GB value = 0.0286 GB dict = 0.99 MB all = 0.0342 GB } UnZip/Zip{ index = 10.5126 value = 3.4705 dict = 1.00 all = 3.5991 } Zip/UnZip{ index = 0.0951 value = 0.2881 dict = 1.00 all = 0.2778 } ---------------------------- total value len = 2253.611947 GB avg = 0.199 KB (by entry num) total key len = 307.398642 GB avg = 0.027 KB total ukey len = 237.723070 GB avg = 0.015 KB total ukey num = 15.757078298 Billion total entry num = 11.298392246 Billion write speed all = 35.163809791 MB/sec (with seq num) write speed all = 33.922754350 MB/sec (without seq num) 2024/11/24-05:09:03.128033 139957643191872 [compaction_job.cc:1948] [default] [JOB 41981] Generated table #293024: 850290 keys, 34173954 bytes, temperature: kWarm 2024/11/24-05:09:03.128104 139957643191872 EVENT_LOG_v1 {"time_micros": 1732396143128068, "cf_name": "default", "job": 41981, "event": "table_file_creation", "file_number": 293024, "file_size": 34173954, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178719, "largest_seqno": 4452999147, "table_properties": {"data_size": 28580544, "index_size": 1617664, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 23808120, "raw_average_key_size": 28, "raw_value_size": 99187612, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 850290, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732396140, "oldest_key_time": 0, "file_creation_time": 1732396140, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 293024, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-05:09:03.158732 139957643191872 [compaction_job.cc:703] job-41981: subcompact[0], size: 0.723161 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-41981/att-00 [293005,293006,293007,293008,293009,293010,293011,293012,293013,293014,293015,293016,293017,293018,293019,293020,293021,293022,293023,293024] 2024/11/24-05:09:03.274901 139957643191872 (Original Log Time 2024/11/24-05:07:57.246153) EVENT_LOG_v1 {"time_micros": 1732396077246110, "job": 41981, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [292998], "files_L3": [293001, 293002, 292925, 292926, 292927, 292928, 292929, 292930, 292931, 292932, 292933, 292934, 292935, 292901, 292902, 292903, 292904, 292905, 292906, 292907, 292908, 292909], "score": 0.712343, "input_data_size": 748933920, "oldest_snapshot_seqno": 4453051770} 2024/11/24-05:09:03.274905 139957643191872 (Original Log Time 2024/11/24-05:09:03.273290) [compaction_job.cc:2022] [default] [JOB 41981] Compacted 1@2 + 22@3 files to L3 => 723161214 bytes 2024/11/24-05:09:03.274906 139957643191872 (Original Log Time 2024/11/24-05:09:03.274825) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 20 0 0 0] max score 0.03, MB/sec: 11.4 rd, 11.0 wr, level 3, files in(1, 22) out(20 +0 blob) MB in(27.1, 687.1 +0.0 blob) out(689.7 +0.0 blob), read-write-amplify(51.8) write-amplify(25.4) OK, records in: 18525767, records dropped: 108632 output_compression: Snappy 2024/11/24-05:09:03.274910 139957643191872 (Original Log Time 2024/11/24-05:09:03.274873) EVENT_LOG_v1 {"time_micros": 1732396143274849, "cf": "default", "job": 41981, "event": "compaction_finished", "compaction_time_micros": 65912589, "compaction_time_cpu_micros": 28544281, "output_level": 3, "num_output_files": 20, "total_output_size": 723161214, "num_input_records": 18525767, "num_output_records": 18417135, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 527023079, "file_range_sync_nanos": 0, "file_fsync_nanos": 13747, "file_prepare_write_nanos": 49911, "lsm_state": [0, 0, 0, 20, 0, 0, 0]} 2024/11/24-05:09:03.415606 139957643191872 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732396077,"executesMs":66180,"compactionJobId":41981,"attempt":0,"compactionInputRawBytes":2688559770,"compactionInputZipBytes":748933920,"compactionOutputRawBytes":0,"compactionOutputZipBytes":723161214,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-d26f029e49c1d5245e8e13857d954d5a-3db4083ff3dbad85-00"}}, response = 2024/11/24-05:09:03.416496 139957643191872 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-41981/att-00: olev 3, work 66.180 s, result 5.943 ms, install 1.636 ms, input{raw 2.504 GiB zip 714.239 MiB}