【详述】4月27号,从版本2.3.0升级至2.5.4版本。未升级之前BE内存使用不超过16G。升级之后,开启异步物化视图,大部分刷新在5s。内存上涨至36G。到5月15号,开始出现内存上涨异常。突破36G,上涨至46G,在内存46G时,BE节点日志报错:
exec multi plan fragments failed, errmsg=Memory of process exceed limit. Start execute plan fragment. Used: 48414205816, Limit: 48410415267. Mem usage has exceed the limit of BE。
FE前端查询时,返回:
ERROR 1064 (HY000): Memory of Fragment cb2b1d1d-f44e-11ed-990b-02426ef29be7 exceed limit. QUERY Backend: , fragment: Used: 2913357568, Limit: 2147483648. Mem usage has exceed the limit of single query, You can change the limit by set session variable exec_mem_limit or query_mem_limit.
【背景】禁用页缓存,修改物化视图刷新为1小时定时刷新一次。
【业务影响】
【StarRocks版本】2.5.4
【集群规模】3fe(3 follower)+3be
【机器信息】CPU虚拟核/内存/网卡,be:16C/64G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息,请补充下您的联系方式,例如:社区群4-小李或者邮箱,谢谢
【附件】
be节点配置:
default_rowset_type = beta
cumulative_compaction_num_threads_per_disk = 4
base_compaction_num_threads_per_disk = 2
cumulative_compaction_check_interval_seconds = 2
routine_load_thread_pool_size = 40
cumulative_compaction_budgeted_bytes=314572800
brpc_max_body_size = 8589934592
mem_limit = 80%
disable_storage_page_cache = true
是查询物化视图时 报这个错么,目前设置为1小时刷新一次 内存降下来了么
设置改为1小时后,问题依旧存在。
curl -XGET -s http://BE_IP:BE_HTTP_PORT/metrics | grep “^starrocks_be_.*_mem_bytes” 发一下当前be内存占用情况, 可以参考 https://docs.starrocks.io/zh-cn/latest/administration/Memory_management#查看内存使用
starrocks_be_bitmap_index_mem_bytes 0
starrocks_be_bloom_filter_index_mem_bytes 0
starrocks_be_chunk_allocator_mem_bytes 2148977696
starrocks_be_clone_mem_bytes 0
starrocks_be_column_metadata_mem_bytes 460156272
starrocks_be_column_pool_mem_bytes 1156992413
starrocks_be_column_zonemap_index_mem_bytes 9948784
starrocks_be_compaction_mem_bytes 2469832
starrocks_be_consistency_mem_bytes 0
starrocks_be_load_mem_bytes 0
starrocks_be_metadata_mem_bytes 692403847
starrocks_be_ordinal_index_mem_bytes 231138416
starrocks_be_process_mem_bytes 26249923224
starrocks_be_query_mem_bytes 16712664
starrocks_be_rowset_metadata_mem_bytes 145440610
starrocks_be_schema_change_mem_bytes 0
starrocks_be_segment_metadata_mem_bytes 19138856
starrocks_be_segment_zonemap_mem_bytes 5082348
starrocks_be_short_key_index_mem_bytes 4403818
starrocks_be_storage_page_cache_mem_bytes 0
starrocks_be_tablet_metadata_mem_bytes 67668109
starrocks_be_tablet_schema_mem_bytes 9788509
starrocks_be_update_mem_bytes 3834244397
中午对be节点重启过一次。这些内存还未达到峰值48G。
starrocks_be_bitmap_index_mem_bytes 0
starrocks_be_bloom_filter_index_mem_bytes 0
starrocks_be_chunk_allocator_mem_bytes 2148977696
starrocks_be_clone_mem_bytes 0
starrocks_be_column_metadata_mem_bytes 460156272
starrocks_be_column_pool_mem_bytes 1156992413
starrocks_be_column_zonemap_index_mem_bytes 9948784
starrocks_be_compaction_mem_bytes 2469832
starrocks_be_consistency_mem_bytes 0
starrocks_be_load_mem_bytes 0
starrocks_be_metadata_mem_bytes 692403847
starrocks_be_ordinal_index_mem_bytes 231138416
starrocks_be_process_mem_bytes 26249923224
starrocks_be_query_mem_bytes 16712664
starrocks_be_rowset_metadata_mem_bytes 145440610
starrocks_be_schema_change_mem_bytes 0
starrocks_be_segment_metadata_mem_bytes 19138856
starrocks_be_segment_zonemap_mem_bytes 5082348
starrocks_be_short_key_index_mem_bytes 4403818
starrocks_be_storage_page_cache_mem_bytes 0
starrocks_be_tablet_metadata_mem_bytes 67668109
starrocks_be_tablet_schema_mem_bytes 9788509
starrocks_be_update_mem_bytes 3834244397
http://be_ip:8040/mem_tracker 和 http://be_ip:8040/memz 这两个的统计信息也帮忙提供下