2024/11/24-02:59:30.869803 139957315876416 DEBG dcompact_worker.cpp:2214: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00: fork to child time = 0.007711 sec 2024/11/24-02:59:30.870530 139957315876416 DEBG dcompact_worker.cpp:1013: Beg SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00 2024/11/24-02:59:30.875948 139957315876416 DEBG dcompact_worker.cpp:1033: End SerDeRead: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00 2024/11/24-02:59:30.876843 139957315876416 TRAC dcompact_worker.cpp:1083: INFO_LOG_LEVEL: rpc = INFO_LEVEL, env_var = undefined 2024/11/24-02:59:30.877208 139957315876416 [WARN] [column_family.cc:386] multiple cf_paths/db_paths and level_compaction_dynamic_level_bytes can't be used together 2024/11/24-02:59:30.929507 139957315876416 [version_set.cc:6208] Recovering from manifest file: /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40477/att-00/MANIFEST-206347 2024/11/24-02:59:30.960275 139957315876416 [version_set.cc:6251] Recovered from manifest file:/worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40477/att-00/MANIFEST-206347 succeeded,manifest_file_number is 206347, next_file_number is 289607, last_sequence is 4407160178, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 289429 2024/11/24-02:59:30.960280 139957315876416 [version_set.cc:6266] Column family [default] (ID 0), log number is 0 2024/11/24-02:59:30.960476 139957315876416 [version_set.cc:5721] Creating manifest 289607 2024/11/24-02:59:30.998172 139957315876416 DEBG dcompact_worker.cpp:1218: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00: bottommost_level: fake = 1, rpc = 0 2024/11/24-02:59:31.043887 139957315876416 [compaction_job.cc:2437] [default] [JOB 40477] Compacting 1@2 + 12@3 files to L3, score 0.90, subcompactions 1 : 1 2024/11/24-02:59:31.043902 139957315876416 [compaction_job.cc:2445] [default]: Compaction start summary: Base version 30320 Base level 2, inputs: [289567(18MB)], [289504(33MB) 289505(35MB) 289506(35MB) 289507(35MB) 289508(34MB) 289509(33MB) 289510(34MB) 289511(35MB) 289512(35MB) 289513(3349KB) 289484(35MB) 289589(34MB)] 2024/11/24-02:59:31.049494 139957315876416 [top_zip_table.cc:59] core git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-02:59:31.049525 139957315876416 [top_zip_table.cc:60] fsa git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-02:59:31.049542 139957315876416 [top_zip_table.cc:61] zbs git_version_hash_info_is:commit fc4ab4f2fbfdcdfa23a955d69e26d2fc533b902d Author: leipeng Date: 2024-04-16 20:29:43 +0800 test_sso.cpp: test swap g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. INCS = -Isrc -I3rdparty/re2 -I3rdparty/zstd -Iboost-include CXXFLAGS = -D TOPLING_IO_WITH_URING=1 -fPIC -time -Wno-class-memaccess -fdiagnostics-color -Wformat=2 -Wcomment -Wall -Wextra -Wno-unused-parameter -Wno-alloc-size-larger-than -mcx16 -Wno-deprecated-declarations -Wstrict-aliasing=3 -DCLIENT_LICENSE_ID="public" -DNO_THREADS -ftls-model=initial-exec -DDIVSUFSORT_API= -fno-stack-protector RLS_FLAGS = -O3 -DNDEBUG -g3 WITH_BMI2 = 1 compile_cpu_flag: -march=haswell -mbmi -mbmi2 2024/11/24-02:59:31.049547 139957315876416 [top_zip_table.cc:62] topling-zip_table_reader: git_version_hash_info_is: commit a4a382006cb82713acf5b3425835f22711877681 Author: leipeng Date: 2024-04-11 19:44:14 +0800 fixed_len_hole_index.cc: Add env bool FixedLenHoleIndexUseAVX512 for ut & debug 1. FixedLenHoleIndexUseAVX512 is only in unit tests or debug 2. In release mode, use avx512 by compile options(-march=) g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-02:59:31.049558 139957315876416 [top_zip_table.cc:64] topling-rocks git_version_hash_info_is: commit feab587e468621ba0204d47b4ffacc52e7c4290d Author: leipeng Date: 2024-03-22 13:35:06 +0800 ToplingZipTableBuilder::Add: immediately return after error This function returns void, and caller does not check builder->status() after calling this function and graceful abandon. So immediately return after error, and return error status in Finish(). g++ (GCC) 12.1.0 Copyright (C) 2022 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. cpu_flag: -march=native -mbmi -mbmi2 2024/11/24-02:59:31.489404 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289608.sst) /dev/shm/ToplingTemp/Topling-253692-5oO6Md 2024/11/24-02:59:32.080412 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289608.sst): first pass time = 0.59's, 227.117'MB/sec 2024/11/24-02:59:34.088561 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289608.sst): old prealloc_size = 147639500, real_size = 31451952 2024/11/24-02:59:34.089273 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289608.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 920306 prefix = 4 raw-key = 0.0147 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1084 GB zip-val = 0.0295 GB avg-val = 117.84 avg-zval = 32.01 2024/11/24-02:59:34.090173 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289608.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:59:35.129526 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289608.sst): second pass time = 2.01's, 54.027'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 1.04's, 34.133'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2556.639'MB/sec (index lex order gen) rebuild zvType time = 0.84's, 0.275'MB/sec write SST data time = 0.20's, 175.674'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.084'MB, throughput = 35514.544'MB/sec zip my value time = 2.01's, unzip length = 0.108'GB zip my value throughput = 54.027'MB/sec zip pipeline throughput = 134.344'MB/sec entries = 920306 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.84 avg-zval = 32.01 usrkeys = 920306 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.84 avg-zval = 32.01 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 920306 bytes = 3221088 } seq expand size = 6442142 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0295 GB dict = 1.08 MB all = 0.0355 GB } UnZip/Zip{ index = 10.4447 value = 3.6813 dict = 1.00 all = 3.7775 } Zip/UnZip{ index = 0.0957 value = 0.2716 dict = 1.00 all = 0.2647 } ---------------------------- total value len = 2251.440465 GB avg = 0.200 KB (by entry num) total key len = 306.908731 GB avg = 0.027 KB total ukey len = 237.373133 GB avg = 0.015 KB total ukey num = 15.739581469 Billion total entry num = 11.280895417 Billion write speed all = 39.321158617 MB/sec (with seq num) write speed all = 37.934083299 MB/sec (without seq num) 2024/11/24-02:59:35.178379 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289608: 920306 keys, 35531116 bytes, temperature: kWarm 2024/11/24-02:59:35.178480 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388375178429, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289608, "file_size": 35531116, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4259662629, "largest_seqno": 4406791003, "table_properties": {"data_size": 29459424, "index_size": 1762240, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25768568, "raw_average_key_size": 28, "raw_value_size": 108449228, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 920306, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388371, "oldest_key_time": 0, "file_creation_time": 1732388371, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289608, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:59:35.215773 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289609.sst) /dev/shm/ToplingTemp/Topling-253692-R8oqBd 2024/11/24-02:59:36.083588 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289609.sst): first pass time = 0.87's, 154.667'MB/sec 2024/11/24-02:59:37.696948 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289609.sst): old prealloc_size = 147639500, real_size = 33081760 2024/11/24-02:59:37.697532 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289609.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 925530 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 117.02 avg-zval = 33.59 2024/11/24-02:59:37.698305 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289609.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:59:38.571994 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289609.sst): second pass time = 1.61's, 67.163'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.88's, 42.487'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3219.608'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.14's, 268.535'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34463.124'MB/sec zip my value time = 1.61's, unzip length = 0.108'GB zip my value throughput = 67.163'MB/sec zip pipeline throughput = 130.410'MB/sec entries = 925530 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.02 avg-zval = 33.59 usrkeys = 925530 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.02 avg-zval = 33.59 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 925530 bytes = 3239376 } seq expand size = 6478710 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5132 value = 3.4836 dict = 1.00 all = 3.6103 } Zip/UnZip{ index = 0.0951 value = 0.2871 dict = 1.00 all = 0.2770 } ---------------------------- total value len = 2251.555246 GB avg = 0.200 KB (by entry num) total key len = 306.934646 GB avg = 0.027 KB total ukey len = 237.391644 GB avg = 0.015 KB total ukey num = 15.740506999 Billion total entry num = 11.281820947 Billion write speed all = 39.321240614 MB/sec (with seq num) write speed all = 37.934124887 MB/sec (without seq num) 2024/11/24-02:59:38.646754 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289609: 925530 keys, 37175918 bytes, temperature: kWarm 2024/11/24-02:59:38.646800 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388378646773, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289609, "file_size": 37175918, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4259662645, "largest_seqno": 4406795351, "table_properties": {"data_size": 31089488, "index_size": 1760704, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25914840, "raw_average_key_size": 28, "raw_value_size": 108302952, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 925530, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388375, "oldest_key_time": 0, "file_creation_time": 1732388375, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289609, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:59:38.689505 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289610.sst) /dev/shm/ToplingTemp/Topling-253692-wClJua 2024/11/24-02:59:39.577171 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289610.sst): first pass time = 0.89's, 151.209'MB/sec 2024/11/24-02:59:41.153954 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289610.sst): old prealloc_size = 147639500, real_size = 33462272 2024/11/24-02:59:41.154491 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289610.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926574 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.92 raw-val = 0.1083 GB zip-val = 0.0315 GB avg-val = 116.85 avg-zval = 33.95 2024/11/24-02:59:41.155248 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289610.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:59:42.013955 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289610.sst): second pass time = 1.58's, 68.696'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.86's, 43.677'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3507.978'MB/sec (index lex order gen) rebuild zvType time = 0.73's, 0.316'MB/sec write SST data time = 0.13's, 298.143'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 28180.884'MB/sec zip my value time = 1.58's, unzip length = 0.108'GB zip my value throughput = 68.696'MB/sec zip pipeline throughput = 134.576'MB/sec entries = 926574 avg-key = 28.00 avg-zkey = 1.92 avg-val = 116.85 avg-zval = 33.95 usrkeys = 926574 avg-key = 20.00 avg-zkey = 1.92 avg-val = 123.85 avg-zval = 33.95 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926574 bytes = 3243024 } seq expand size = 6486018 multi value expand size = 0 cnt WriteAppend = 14 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0315 GB dict = 1.08 MB all = 0.0376 GB } UnZip/Zip{ index = 10.4430 value = 3.4421 dict = 1.00 all = 3.5734 } Zip/UnZip{ index = 0.0958 value = 0.2905 dict = 1.00 all = 0.2798 } ---------------------------- total value len = 2251.670006 GB avg = 0.200 KB (by entry num) total key len = 306.960590 GB avg = 0.027 KB total ukey len = 237.410176 GB avg = 0.015 KB total ukey num = 15.741433573 Billion total entry num = 11.282747521 Billion write speed all = 39.321323013 MB/sec (with seq num) write speed all = 37.934166741 MB/sec (without seq num) 2024/11/24-02:59:42.061160 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289610: 926574 keys, 37560164 bytes, temperature: kWarm 2024/11/24-02:59:42.061230 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388382061194, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289610, "file_size": 37560164, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178503, "largest_seqno": 4406793915, "table_properties": {"data_size": 31455920, "index_size": 1774528, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944072, "raw_average_key_size": 28, "raw_value_size": 108273656, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926574, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388378, "oldest_key_time": 0, "file_creation_time": 1732388378, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289610, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:59:42.099623 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289611.sst) /dev/shm/ToplingTemp/Topling-253692-IW9a3d 2024/11/24-02:59:42.968939 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289611.sst): first pass time = 0.87's, 154.400'MB/sec 2024/11/24-02:59:44.504406 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289611.sst): old prealloc_size = 147639500, real_size = 33107616 2024/11/24-02:59:44.504939 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289611.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926549 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0311 GB avg-val = 116.86 avg-zval = 33.58 2024/11/24-02:59:44.505700 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289611.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:59:45.354396 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289611.sst): second pass time = 1.53's, 70.552'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 43.773'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3440.297'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.320'MB/sec write SST data time = 0.13's, 294.737'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 35877.943'MB/sec zip my value time = 1.53's, unzip length = 0.108'GB zip my value throughput = 70.552'MB/sec zip pipeline throughput = 113.090'MB/sec entries = 926549 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 33.58 usrkeys = 926549 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 33.58 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926549 bytes = 3242944 } seq expand size = 6485843 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0311 GB dict = 1.08 MB all = 0.0372 GB } UnZip/Zip{ index = 10.5129 value = 3.4800 dict = 1.00 all = 3.6075 } Zip/UnZip{ index = 0.0951 value = 0.2874 dict = 1.00 all = 0.2772 } ---------------------------- total value len = 2251.784766 GB avg = 0.200 KB (by entry num) total key len = 306.986533 GB avg = 0.027 KB total ukey len = 237.428707 GB avg = 0.015 KB total ukey num = 15.742360122 Billion total entry num = 11.283674070 Billion write speed all = 39.321466731 MB/sec (with seq num) write speed all = 37.934267758 MB/sec (without seq num) 2024/11/24-02:59:45.402652 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289611: 926549 keys, 37205288 bytes, temperature: kWarm 2024/11/24-02:59:45.402715 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388385402680, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289611, "file_size": 37205288, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178518, "largest_seqno": 4406795087, "table_properties": {"data_size": 31113104, "index_size": 1762688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25943372, "raw_average_key_size": 28, "raw_value_size": 108274390, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926549, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388382, "oldest_key_time": 0, "file_creation_time": 1732388382, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289611, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:59:45.438115 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289612.sst) /dev/shm/ToplingTemp/Topling-253692-8O86se 2024/11/24-02:59:46.291440 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289612.sst): first pass time = 0.85's, 157.295'MB/sec 2024/11/24-02:59:47.839601 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289612.sst): old prealloc_size = 147639500, real_size = 31578656 2024/11/24-02:59:47.840154 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289612.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926579 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0296 GB avg-val = 116.85 avg-zval = 31.93 2024/11/24-02:59:47.840905 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289612.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:59:48.668724 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289612.sst): second pass time = 1.55's, 70.002'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 43.030'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3322.516'MB/sec (index lex order gen) rebuild zvType time = 0.70's, 0.331'MB/sec write SST data time = 0.13's, 278.321'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 39034.249'MB/sec zip my value time = 1.55's, unzip length = 0.108'GB zip my value throughput = 70.002'MB/sec zip pipeline throughput = 111.532'MB/sec entries = 926579 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.85 avg-zval = 31.93 usrkeys = 926579 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.85 avg-zval = 31.93 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926579 bytes = 3243040 } seq expand size = 6486053 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0296 GB dict = 1.08 MB all = 0.0357 GB } UnZip/Zip{ index = 10.5133 value = 3.6599 dict = 1.00 all = 3.7621 } Zip/UnZip{ index = 0.0951 value = 0.2732 dict = 1.00 all = 0.2658 } ---------------------------- total value len = 2251.899526 GB avg = 0.200 KB (by entry num) total key len = 307.012478 GB avg = 0.027 KB total ukey len = 237.447238 GB avg = 0.015 KB total ukey num = 15.743286701 Billion total entry num = 11.284600649 Billion write speed all = 39.321626224 MB/sec (with seq num) write speed all = 37.934383995 MB/sec (without seq num) 2024/11/24-02:59:48.713230 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289612: 926579 keys, 35676324 bytes, temperature: kWarm 2024/11/24-02:59:48.713295 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388388713263, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289612, "file_size": 35676324, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178530, "largest_seqno": 4406794980, "table_properties": {"data_size": 29584144, "index_size": 1762688, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25944212, "raw_average_key_size": 28, "raw_value_size": 108273614, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926579, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388385, "oldest_key_time": 0, "file_creation_time": 1732388385, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289612, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:59:48.753986 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289613.sst) /dev/shm/ToplingTemp/Topling-253692-CfEM3d 2024/11/24-02:59:49.634839 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289613.sst): first pass time = 0.88's, 152.380'MB/sec 2024/11/24-02:59:51.129620 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289613.sst): old prealloc_size = 147639500, real_size = 31268784 2024/11/24-02:59:51.130204 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289613.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 922033 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.93 raw-val = 0.1084 GB zip-val = 0.0293 GB avg-val = 117.57 avg-zval = 31.74 2024/11/24-02:59:51.130932 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289613.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:59:51.941915 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289613.sst): second pass time = 1.49's, 72.561'MB/sec, value only(80.8% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 43.524'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3175.216'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.336'MB/sec write SST data time = 0.12's, 283.078'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.083'MB, throughput = 24055.823'MB/sec zip my value time = 1.49's, unzip length = 0.108'GB zip my value throughput = 72.561'MB/sec zip pipeline throughput = 121.053'MB/sec entries = 922033 avg-key = 28.00 avg-zkey = 1.93 avg-val = 117.57 avg-zval = 31.74 usrkeys = 922033 avg-key = 20.00 avg-zkey = 1.93 avg-val = 124.57 avg-zval = 31.74 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 922033 bytes = 3227136 } seq expand size = 6454231 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0184 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0293 GB dict = 1.08 MB all = 0.0354 GB } UnZip/Zip{ index = 10.3762 value = 3.7046 dict = 1.00 all = 3.7965 } Zip/UnZip{ index = 0.0964 value = 0.2699 dict = 1.00 all = 0.2634 } ---------------------------- total value len = 2252.014381 GB avg = 0.200 KB (by entry num) total key len = 307.038294 GB avg = 0.027 KB total ukey len = 237.465679 GB avg = 0.015 KB total ukey num = 15.744208734 Billion total entry num = 11.285522682 Billion write speed all = 39.321810082 MB/sec (with seq num) write speed all = 37.934524282 MB/sec (without seq num) 2024/11/24-02:59:51.985546 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289613: 922033 keys, 35352810 bytes, temperature: kWarm 2024/11/24-02:59:51.985604 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388391985573, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289613, "file_size": 35352810, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4270052431, "largest_seqno": 4406791055, "table_properties": {"data_size": 29260896, "index_size": 1777216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25816924, "raw_average_key_size": 28, "raw_value_size": 108400902, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 922033, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388388, "oldest_key_time": 0, "file_creation_time": 1732388388, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289613, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:59:52.022696 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289614.sst) /dev/shm/ToplingTemp/Topling-253692-lokM3d 2024/11/24-02:59:52.870154 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289614.sst): first pass time = 0.85's, 158.383'MB/sec 2024/11/24-02:59:54.470037 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289614.sst): old prealloc_size = 147639500, real_size = 32725744 2024/11/24-02:59:54.470555 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289614.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 923759 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1084 GB zip-val = 0.0307 GB avg-val = 117.30 avg-zval = 33.27 2024/11/24-02:59:54.471299 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289614.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:59:55.303374 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289614.sst): second pass time = 1.60's, 67.759'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.83's, 44.181'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3582.144'MB/sec (index lex order gen) rebuild zvType time = 0.71's, 0.326'MB/sec write SST data time = 0.12's, 296.129'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 18925.832'MB/sec zip my value time = 1.60's, unzip length = 0.108'GB zip my value throughput = 67.759'MB/sec zip pipeline throughput = 133.932'MB/sec entries = 923759 avg-key = 28.00 avg-zkey = 1.90 avg-val = 117.30 avg-zval = 33.27 usrkeys = 923759 avg-key = 20.00 avg-zkey = 1.90 avg-val = 124.30 avg-zval = 33.27 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 923759 bytes = 3233168 } seq expand size = 6466313 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1084 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0307 GB dict = 1.08 MB all = 0.0368 GB } UnZip/Zip{ index = 10.5148 value = 3.5251 dict = 1.00 all = 3.6456 } Zip/UnZip{ index = 0.0951 value = 0.2837 dict = 1.00 all = 0.2743 } ---------------------------- total value len = 2252.129200 GB avg = 0.200 KB (by entry num) total key len = 307.064160 GB avg = 0.027 KB total ukey len = 237.484154 GB avg = 0.015 KB total ukey num = 15.745132493 Billion total entry num = 11.286446441 Billion write speed all = 39.321940767 MB/sec (with seq num) write speed all = 37.934613071 MB/sec (without seq num) 2024/11/24-02:59:55.348917 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289614: 923759 keys, 36816038 bytes, temperature: kWarm 2024/11/24-02:59:55.348991 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388395348942, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289614, "file_size": 36816038, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178548, "largest_seqno": 4406793094, "table_properties": {"data_size": 30737568, "index_size": 1757056, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25865252, "raw_average_key_size": 28, "raw_value_size": 108352530, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 923759, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388392, "oldest_key_time": 0, "file_creation_time": 1732388392, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289614, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:59:55.384473 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289615.sst) /dev/shm/ToplingTemp/Topling-253692-GcWj4b 2024/11/24-02:59:56.246898 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289615.sst): first pass time = 0.86's, 155.635'MB/sec 2024/11/24-02:59:57.912093 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289615.sst): old prealloc_size = 147639500, real_size = 33434784 2024/11/24-02:59:57.912641 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289615.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926437 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0314 GB avg-val = 116.88 avg-zval = 33.94 2024/11/24-02:59:57.913427 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289615.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-02:59:58.778459 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289615.sst): second pass time = 1.66's, 65.055'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 43.321'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3340.490'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.314'MB/sec write SST data time = 0.13's, 293.690'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 34171.548'MB/sec zip my value time = 1.66's, unzip length = 0.108'GB zip my value throughput = 65.055'MB/sec zip pipeline throughput = 121.997'MB/sec entries = 926437 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.88 avg-zval = 33.94 usrkeys = 926437 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.88 avg-zval = 33.94 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926437 bytes = 3242544 } seq expand size = 6485059 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0314 GB dict = 1.08 MB all = 0.0375 GB } UnZip/Zip{ index = 10.5128 value = 3.4439 dict = 1.00 all = 3.5761 } Zip/UnZip{ index = 0.0951 value = 0.2904 dict = 1.00 all = 0.2796 } ---------------------------- total value len = 2252.243962 GB avg = 0.200 KB (by entry num) total key len = 307.090100 GB avg = 0.027 KB total ukey len = 237.502683 GB avg = 0.015 KB total ukey num = 15.746058930 Billion total entry num = 11.287372878 Billion write speed all = 39.322003075 MB/sec (with seq num) write speed all = 37.934635579 MB/sec (without seq num) 2024/11/24-02:59:58.825098 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289615: 926437 keys, 37531908 bytes, temperature: kWarm 2024/11/24-02:59:58.825157 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388398825125, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289615, "file_size": 37531908, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178559, "largest_seqno": 4406794570, "table_properties": {"data_size": 31440464, "index_size": 1762496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25940236, "raw_average_key_size": 28, "raw_value_size": 108277534, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926437, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388395, "oldest_key_time": 0, "file_creation_time": 1732388395, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289615, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-02:59:58.861795 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289616.sst) /dev/shm/ToplingTemp/Topling-253692-HzRiXa 2024/11/24-02:59:59.743569 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289616.sst): first pass time = 0.88's, 152.220'MB/sec 2024/11/24-03:00:01.300444 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289616.sst): old prealloc_size = 147639500, real_size = 33338784 2024/11/24-03:00:01.301023 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289616.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926522 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0313 GB avg-val = 116.86 avg-zval = 33.83 2024/11/24-03:00:01.301802 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289616.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:00:02.170642 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289616.sst): second pass time = 1.56's, 69.582'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.87's, 43.022'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3180.034'MB/sec (index lex order gen) rebuild zvType time = 0.74's, 0.312'MB/sec write SST data time = 0.13's, 295.014'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.081'MB, throughput = 12787.618'MB/sec zip my value time = 1.56's, unzip length = 0.108'GB zip my value throughput = 69.582'MB/sec zip pipeline throughput = 116.408'MB/sec entries = 926522 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.86 avg-zval = 33.83 usrkeys = 926522 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.86 avg-zval = 33.83 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926522 bytes = 3242848 } seq expand size = 6485654 multi value expand size = 0 cnt WriteAppend = 11 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0313 GB dict = 1.08 MB all = 0.0374 GB } UnZip/Zip{ index = 10.5141 value = 3.4544 dict = 1.00 all = 3.5852 } Zip/UnZip{ index = 0.0951 value = 0.2895 dict = 1.00 all = 0.2789 } ---------------------------- total value len = 2252.358723 GB avg = 0.200 KB (by entry num) total key len = 307.116043 GB avg = 0.027 KB total ukey len = 237.521213 GB avg = 0.015 KB total ukey num = 15.746985452 Billion total entry num = 11.288299400 Billion write speed all = 39.322115479 MB/sec (with seq num) write speed all = 37.934706410 MB/sec (without seq num) 2024/11/24-03:00:02.218106 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289616: 926522 keys, 37436512 bytes, temperature: kWarm 2024/11/24-03:00:02.218177 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388402218142, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289616, "file_size": 37436512, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178575, "largest_seqno": 4406794251, "table_properties": {"data_size": 31344528, "index_size": 1762432, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25942616, "raw_average_key_size": 28, "raw_value_size": 108275176, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926522, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388398, "oldest_key_time": 0, "file_creation_time": 1732388398, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289616, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:00:02.270357 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289617.sst) /dev/shm/ToplingTemp/Topling-253692-XOtghe 2024/11/24-03:00:03.265426 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289617.sst): first pass time = 1.00's, 134.891'MB/sec 2024/11/24-03:00:04.713998 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289617.sst): old prealloc_size = 147639500, real_size = 32353520 2024/11/24-03:00:04.714566 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289617.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 926459 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.90 raw-val = 0.1083 GB zip-val = 0.0304 GB avg-val = 116.87 avg-zval = 32.77 2024/11/24-03:00:04.716378 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289617.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:00:05.564542 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289617.sst): second pass time = 1.45's, 74.789'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.85's, 42.856'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3241.169'MB/sec (index lex order gen) rebuild zvType time = 0.72's, 0.321'MB/sec write SST data time = 0.13's, 282.893'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.080'MB, throughput = 30302.286'MB/sec zip my value time = 1.45's, unzip length = 0.108'GB zip my value throughput = 74.789'MB/sec zip pipeline throughput = 74.789'MB/sec entries = 926459 avg-key = 28.00 avg-zkey = 1.90 avg-val = 116.87 avg-zval = 32.77 usrkeys = 926459 avg-key = 20.00 avg-zkey = 1.90 avg-val = 123.87 avg-zval = 32.77 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 926459 bytes = 3242624 } seq expand size = 6485213 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0304 GB dict = 1.08 MB all = 0.0365 GB } UnZip/Zip{ index = 10.5130 value = 3.5665 dict = 1.00 all = 3.6822 } Zip/UnZip{ index = 0.0951 value = 0.2804 dict = 1.00 all = 0.2716 } ---------------------------- total value len = 2252.473485 GB avg = 0.200 KB (by entry num) total key len = 307.141983 GB avg = 0.027 KB total ukey len = 237.539742 GB avg = 0.015 KB total ukey num = 15.747911911 Billion total entry num = 11.289225859 Billion write speed all = 39.322226819 MB/sec (with seq num) write speed all = 37.934776227 MB/sec (without seq num) 2024/11/24-03:00:05.612680 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289617: 926459 keys, 36450812 bytes, temperature: kWarm 2024/11/24-03:00:05.612755 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388405612717, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289617, "file_size": 36450812, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178589, "largest_seqno": 4406794800, "table_properties": {"data_size": 30359200, "index_size": 1762496, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25940852, "raw_average_key_size": 28, "raw_value_size": 108276994, "raw_average_value_size": 116, "num_data_blocks": 1, "num_entries": 926459, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388402, "oldest_key_time": 0, "file_creation_time": 1732388402, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289617, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:00:05.650183 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289618.sst) /dev/shm/ToplingTemp/Topling-253692-dpTU1b 2024/11/24-03:00:06.600257 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289618.sst): first pass time = 0.95's, 141.277'MB/sec 2024/11/24-03:00:08.020437 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289618.sst): old prealloc_size = 147639500, real_size = 30961664 2024/11/24-03:00:08.020964 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289618.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 924234 prefix = 4 raw-key = 0.0148 GB zip-key = 0.0018 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.1083 GB zip-val = 0.0290 GB avg-val = 117.22 avg-zval = 31.34 2024/11/24-03:00:08.021760 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289618.sst: sumWaitingMem = 0.002 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0023 GB, waited 0.000 sec, Key+Value = 0.134 GB 2024/11/24-03:00:08.833421 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289618.sst): second pass time = 1.42's, 76.327'MB/sec, value only(80.7% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.81's, 42.973'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 3431.541'MB/sec (index lex order gen) rebuild zvType time = 0.69's, 0.337'MB/sec write SST data time = 0.13's, 274.717'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 1.082'MB, throughput = 36733.007'MB/sec zip my value time = 1.42's, unzip length = 0.108'GB zip my value throughput = 76.327'MB/sec zip pipeline throughput = 76.327'MB/sec entries = 924234 avg-key = 28.00 avg-zkey = 1.91 avg-val = 117.22 avg-zval = 31.34 usrkeys = 924234 avg-key = 20.00 avg-zkey = 1.91 avg-val = 124.22 avg-zval = 31.34 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 924234 bytes = 3119312 } seq expand size = 6469638 multi value expand size = 0 cnt WriteAppend = 12 UnZipSize{ index = 0.0185 GB value = 0.1083 GB dict = 1.08 MB all = 0.1342 GB } __ZipSize{ index = 0.0018 GB value = 0.0290 GB dict = 1.08 MB all = 0.0349 GB } UnZip/Zip{ index = 10.4461 value = 3.7409 dict = 1.00 all = 3.8418 } Zip/UnZip{ index = 0.0957 value = 0.2673 dict = 1.00 all = 0.2603 } ---------------------------- total value len = 2252.588294 GB avg = 0.200 KB (by entry num) total key len = 307.167862 GB avg = 0.027 KB total ukey len = 237.558227 GB avg = 0.015 KB total ukey num = 15.748836145 Billion total entry num = 11.290150093 Billion write speed all = 39.322413433 MB/sec (with seq num) write speed all = 37.934918929 MB/sec (without seq num) 2024/11/24-03:00:08.878267 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289618: 924234 keys, 34935838 bytes, temperature: kWarm 2024/11/24-03:00:08.878336 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388408878301, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289618, "file_size": 34935838, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4283384070, "largest_seqno": 4406791115, "table_properties": {"data_size": 28960880, "index_size": 1769536, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 25878552, "raw_average_key_size": 28, "raw_value_size": 108339300, "raw_average_value_size": 117, "num_data_blocks": 1, "num_entries": 924234, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388405, "oldest_key_time": 0, "file_creation_time": 1732388405, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289618, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:00:08.917089 139957315876416 [top_zip_table_builder.cc:934] ToplingZipTableBuilder(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289619.sst) /dev/shm/ToplingTemp/Topling-253692-fshj7b 2024/11/24-03:00:09.009509 139957315876416 [top_zip_table_builder.cc:1352] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289619.sst): first pass time = 0.09's, 239.471'MB/sec 2024/11/24-03:00:09.236130 139957315876416 [top_zip_table_builder.cc:2772] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289619.sst): old prealloc_size = 147639500, real_size = 6496176 2024/11/24-03:00:09.236258 139957315876416 [top_zip_table_builder.cc:1773] BuildReorderMap(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289619.sst): index type = NestLoudsTrieDAWG_Mixed_XL_256_32_FL, store type = DictZipBlobStore usrkeys = 153797 prefix = 4 raw-key = 0.0025 GB zip-key = 0.0003 GB avg-key = 16.00 avg-zkey = 1.91 raw-val = 0.0178 GB zip-val = 0.0062 GB avg-val = 115.85 avg-zval = 40.08 2024/11/24-03:00:09.236424 139957315876416 [top_zip_table_builder.cc:1299] /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289619.sst: sumWaitingMem = 0.000 GB, sumWorkingMem = 0.006 GB, reorder workingMem = 0.0003 GB, waited 0.000 sec, Key+Value = 0.022 GB 2024/11/24-03:00:09.370146 139957315876416 [top_zip_table_builder.cc:3100] ToplingZipTableBuilder::Finish(/storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00/289619.sst): second pass time = 0.23's, 78.847'MB/sec, value only(80.5% of KV) wait indexing time = 0.00's, remap KeyValue time = 0.13's, 53.627'MB/sec (all stages of remap) Get OrderMap time = 0.00's, 2678.715'MB/sec (index lex order gen) rebuild zvType time = 0.11's, 0.343'MB/sec write SST data time = 0.02's, 329.726'MB/sec dict compress time = 4356233619.02's, 0.000'MB/sec z-dict build time = 0.00's, sample length = 0.187'MB, throughput = 6247.670'MB/sec zip my value time = 0.23's, unzip length = 0.018'GB zip my value throughput = 78.847'MB/sec zip pipeline throughput = 78.847'MB/sec entries = 153797 avg-key = 28.00 avg-zkey = 1.91 avg-val = 115.85 avg-zval = 40.08 usrkeys = 153797 avg-key = 20.00 avg-zkey = 1.91 avg-val = 122.85 avg-zval = 40.08 TagRS{ kinds = RS_Key0_Tag1 bytes = 0 } TagArray{ size = 153797 bytes = 538304 } seq expand size = 1076579 multi value expand size = 0 cnt WriteAppend = 13 UnZipSize{ index = 0.0031 GB value = 0.0178 GB dict = 0.19 MB all = 0.0221 GB } __ZipSize{ index = 0.0003 GB value = 0.0062 GB dict = 0.19 MB all = 0.0072 GB } UnZip/Zip{ index = 10.4595 value = 2.8908 dict = 1.00 all = 3.0791 } Zip/UnZip{ index = 0.0956 value = 0.3459 dict = 1.00 all = 0.3248 } ---------------------------- total value len = 2252.607188 GB avg = 0.200 KB (by entry num) total key len = 307.172168 GB avg = 0.027 KB total ukey len = 237.561303 GB avg = 0.015 KB total ukey num = 15.748989942 Billion total entry num = 11.290303890 Billion write speed all = 39.322445618 MB/sec (with seq num) write speed all = 37.934943654 MB/sec (without seq num) 2024/11/24-03:00:09.386222 139957315876416 [compaction_job.cc:1948] [default] [JOB 40477] Generated table #289619: 153797 keys, 7185119 bytes, temperature: kWarm 2024/11/24-03:00:09.386302 139957315876416 EVENT_LOG_v1 {"time_micros": 1732388409386257, "cf_name": "default", "job": 40477, "event": "table_file_creation", "file_number": 289619, "file_size": 7185119, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 4261178599, "largest_seqno": 4406788989, "table_properties": {"data_size": 6163488, "index_size": 294080, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 0, "raw_key_size": 4306316, "raw_average_key_size": 28, "raw_value_size": 17817306, "raw_average_value_size": 115, "num_data_blocks": 1, "num_entries": 153797, "num_filter_entries": 0, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 28, "fixed_value_len": 18446744073709551615, "filter_policy": "", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "nullptr", "prefix_extractor_name": "nullptr", "property_collectors": "[Rdb_tbl_prop_coll_factory]", "compression": "FlatZip", "compression_options": "RS_Key0_Tag1-NestLoudsTrieDAWG_Mixed_XL_256_32_FL-DictZipBlobStore", "creation_time": 1732388408, "oldest_key_time": 0, "file_creation_time": 1732388408, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d89b4006-2803-486e-b7a2-759bcf348921", "db_session_id": "U1ECJGOXSO4Y1P7ZZXV1", "orig_file_number": 289619, "seqno_to_time_mapping": "N/A", "__indexstats__": "[...1 records...]"}} 2024/11/24-03:00:09.397100 139957315876416 [compaction_job.cc:703] job-40477: subcompact[0], size: 0.408858 G, files: /storage/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/mytopling/.rocksdb/job-40477/att-00 [289608,289609,289610,289611,289612,289613,289614,289615,289616,289617,289618,289619] 2024/11/24-03:00:09.468963 139957315876416 (Original Log Time 2024/11/24-02:59:31.043941) EVENT_LOG_v1 {"time_micros": 1732388371043912, "job": 40477, "event": "compaction_started", "compaction_reason": "FilesMarkedForCompaction", "files_L2": [289567], "files_L3": [289504, 289505, 289506, 289507, 289508, 289509, 289510, 289511, 289512, 289513, 289484, 289589], "score": 0.897483, "input_data_size": 425684736, "oldest_snapshot_seqno": 4407159993} 2024/11/24-03:00:09.468967 139957315876416 (Original Log Time 2024/11/24-03:00:09.467915) [compaction_job.cc:2022] [default] [JOB 40477] Compacted 1@2 + 12@3 files to L3 => 408857847 bytes 2024/11/24-03:00:09.468968 139957315876416 (Original Log Time 2024/11/24-03:00:09.468899) [compaction_job.cc:1224] [default] compacted to: files[0 0 0 12 0 0 0] max score 0.02, MB/sec: 11.1 rd, 10.7 wr, level 3, files in(1, 12) out(12 +0 blob) MB in(18.1, 387.8 +0.0 blob) out(389.9 +0.0 blob), read-write-amplify(43.9) write-amplify(21.5) OK, records in: 10401877, records dropped: 73098 output_compression: Snappy 2024/11/24-03:00:09.468997 139957315876416 (Original Log Time 2024/11/24-03:00:09.468942) EVENT_LOG_v1 {"time_micros": 1732388409468924, "cf": "default", "job": 40477, "event": "compaction_finished", "compaction_time_micros": 38353166, "compaction_time_cpu_micros": 15784931, "output_level": 3, "num_output_files": 12, "total_output_size": 408857847, "num_input_records": 10401877, "num_output_records": 10328779, "num_subcompactions": 1, "output_compression": "Snappy", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "file_write_nanos": 284738533, "file_range_sync_nanos": 0, "file_fsync_nanos": 7930, "file_prepare_write_nanos": 27680, "lsm_state": [0, 0, 0, 12, 0, 0, 0]} 2024/11/24-03:00:09.595596 139957315876416 DEBG dcompact_worker.cpp:240: HttpPost: 200 OK: url = http://nlb-7zny4kaa1poal01o64.cn-shenzhen.nlb.aliyuncs.com:8443, body = {"instanceId":"nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137","labourId":"","dbId":"d89b4006-2803-486e-b7a2-759bcf348921","dbStarts":1732256054,"starts":1732388370,"executesMs":38590,"compactionJobId":40477,"attempt":0,"compactionInputRawBytes":1508899310,"compactionInputZipBytes":425684736,"compactionOutputRawBytes":0,"compactionOutputZipBytes":408857847,"headers":{"Accept":"*/*","Connection":"close","Host":"aliyun-topling-saas.topling.cn:18443","RemoteIp":"120.24.192.169","X-Forwarded-For":"120.24.192.169","region":"cn-shenzhen","token":"e6474c481bd4415abc537135cc03626b","instance-token":"e6474c481bd4415abc537135cc03626b","provider":"aliyun","CloudMarketInstanceId":"si-59c0ae5c955845588903","MeteringAssit":"","Transfer-Encoding":"chunked","traceparent":"00-0e10d87d43b11b49e9b87f1b6dd5826c-1f0b52abd7bfdf20-00"}}, response = 2024/11/24-03:00:09.595677 139957315876416 INFO dcompact_worker.cpp:1459: finish /worker/dcompact-log/nfs/i-wz9deglxdazgiryp6ltb_10.57.5.137/2024-11-22T14.14.14/.rocksdb/job-40477/att-00: olev 3, work 38.590 s, result 7.730 ms, install 1.106 ms, input{raw 1.405 GiB zip 405.965 MiB}