无查询,belog报 large memory alloc Memory limit exceeded

为了更快的定位您的问题,请提供以下信息,谢谢
【详述】
【背景】
【业务影响】
【是否存算分离】是
【StarRocks版本】3.1.4
【集群规模】例如:1fe +2be(fe与be混部)
【表模型】例如:主键模型
【导入或者导出方式】 datax

【附件】
W1206 18:58:15.323366 21969 mem_hook.cpp:266] large memory alloc: 1476395024 bytes, stack:
@ 0x4e766b3 malloc
@ 0x873feb5 operator new()
@ 0x45138bd std::vector<>::_M_range_insert<>()
@ 0x451509e starrocks::PrimaryKeyEncoder::encode()
@ 0x47f431d starrocks::lake::CompactionState::_load_segments()
@ 0x47f4963 starrocks::lake::CompactionState::load_segments()
@ 0x47bb8ee starrocks::lake::UpdateManager::publish_primary_compaction()
@ 0x5949086 starrocks::lake::PrimaryKeyTxnLogApplier::apply()
@ 0x593ecfd starrocks::lake::publish_version()
@ 0x4e8c1cd _ZZN9starrocks15LakeServiceImpl15publish_versionEPN6google8protobuf13RpcControllerEPKNS_4lake21PublishVersionRequestEPNS5_22PublishVersionResponseEPNS2_7ClosureEENKUlvE_clEv
@ 0x4f782b2 starrocks::ThreadPool::dispatch_thread()
@ 0x4f72d4a starrocks::thread::supervise_thread()
@ 0x7f8e6b5eaea5 start_thread
@ 0x7f8e6a9ebb0d __clone
@ (nil) (unknown)

W1206 18:58:35.721650 21968 transactions.cpp:104] Fail to apply staros://22664/log/0000000000005888_000000000000F316.log: Memory limit exceeded: Memory of process exceed limit. read and decompress page Used: 13291217648, Limit: 13263024291. Mem usage has exceed the limit of BE

/build/starrocks/be/src/storage/rowset/page_io.cpp:135 CurrentThread::mem_tracker()->check_mem_limit(“read and decompress page”)

/build/starrocks/be/src/storage/rowset/scalar_column_iterator.cpp:292 _reader->read_page(_opts, iter.page(), &handle, &page_body, &footer)

/build/starrocks/be/src/storage/rowset/scalar_column_iterator.cpp:252 _read_data_page(_page_iter)

/build/starrocks/be/src/storage/rowset/scalar_column_iterator.cpp:201 _load_next_page(&eos)

/build/starrocks/be/src/storage/rowset/segment_iterator.cpp:134 _column_iterators[i]->next_batch(range, col.get())

/build/starrocks/be/src/storage/rowset/segment_iterator.cpp:938 _context->read_columns(chunk, range)

/build/starrocks/be/src/storage/rowset/segment_iterator.cpp:1011 _read(chunk, rowid, chunk_capacity - chunk_start)

/build/starrocks/be/src/storage/lake/update_manager.cpp:524 compaction_state->load_segments(output_rowset.get(), *tablet_schema, i)

/build/starrocks/be/src/storage/lake/txn_log_applier.cpp:81 apply_compaction_log(log.op_compaction())

W1206 18:58:35.721772 21968 update_manager.cpp:568] Lake update manager remove primary index cache, tablet_id: 22664 , succ: 1

starrocks_be_bitmap_index_mem_bytes 0

starrocks_be_bloom_filter_index_mem_bytes 0

starrocks_be_chunk_allocator_mem_bytes 12312

starrocks_be_clone_mem_bytes 0

starrocks_be_column_metadata_mem_bytes 60479280

starrocks_be_column_pool_mem_bytes 0

starrocks_be_column_zonemap_index_mem_bytes 26915840

starrocks_be_compaction_mem_bytes 0

starrocks_be_consistency_mem_bytes 0

starrocks_be_load_mem_bytes 0

starrocks_be_metadata_mem_bytes 76215008

starrocks_be_ordinal_index_mem_bytes 14385904

starrocks_be_process_mem_bytes 12062227888

starrocks_be_query_mem_bytes 0

starrocks_be_rowset_metadata_mem_bytes 0

starrocks_be_schema_change_mem_bytes 0

starrocks_be_segment_metadata_mem_bytes 15722480

starrocks_be_segment_zonemap_mem_bytes 13962136

starrocks_be_short_key_index_mem_bytes 0

starrocks_be_storage_page_cache_mem_bytes 49417312

starrocks_be_tablet_metadata_mem_bytes 13248

starrocks_be_tablet_schema_mem_bytes 13248

starrocks_be_update_mem_bytes 191037069

这是be在做compaction。请问be配置有改过吗?机器的内存是多少呢?

改了mem_limit, 机器内存是16G

已经升级到32G了, 重启后,be日志还有报 2023-12-07 10:56:36,892 WARN (AutoStatistic|27) [StmtExecutor.executeStmtWithExecPlan():2021] com.starrocks.common.UserException: Memory of process exceed limit. Pipeline Backend: 172.16.132.124, fragment: 3bf7917a-94ac-11ee-a80d-00163e354102 Used: 27595463844, Limit: 26745889996. Mem usage has exceed the limit of BE

有查询吗?

没有查询

image 请问 compact version是什么意思, 为啥搞了2000多次呀

compaction version的意思是这个版本是由compaction生成的,不是做了2000多次的意思

现在mem_limit是多少?

Mem Limit: 24.91 GB
Mem Consumption: 18.15 GB

be 配置的mem limit 是 90%

Fail to apply staros://22672/log/0000000000005890_000000000000FA16.log: Memory limit exceeded: Memory of process exceed limit. LoadSegments Used: 33601712096, Limit: 26745889996. Mem usage has exceed the limit of BE
请问这个 staros 的这个log 文件, 具体怎样可以访问

W1206 18:08:03.853889 16575 transactions.cpp:104] Fail to apply staros://22672/log/0000000000005890_000000000000F313.log: Memory limit exceeded: Memory of process exceed limit. LoadSegments Used: 10645378112, Limit: 10315685559. Mem usage has exceed the limit of BE
还有就是这个内存超限制,我的配置问题么,我发现直接是申请10G, 现在我升级完之后,现在都申请 30G。还是超限

可以提供一下be.INFO吗?cpu是多少core呢?

这个文件不太关键的

disable_column_pool=1

compact使用的线程数

compact_threads = 2
compact 造成的内存 占用, 加上上面两个参数就好了。
多谢罗老师和激流的指导

disable_column_pool

compact_threads
这两个参数在官方文档怎么找不到呢?
请问您是在哪里看到的呀