物化视图初始化失败

💬 StarRocks 用户问答 功能使用相关
#1

为了更快的定位您的问题，请提供以下信息，谢谢
【详述】创建一个 基于 hive catalog 的分区物化视图

【背景】做过哪些操作？
【业务影响】
【是否存算分离】 是
【StarRocks版本】3.1.4
【集群规模】1fe + 1be
【机器信息】8核 + 32G
【附件】

W1212 13:04:23.800501 30128 query_context.cpp:610] Retrying ReportExecStatus: No more data to read.

W1212 13:04:53.802970 30128 query_context.cpp:610] Retrying ReportExecStatus: No more data to read.

W1212 13:05:23.805472 30128 query_context.cpp:610] Retrying ReportExecStatus: No more data to read.

W1212 13:05:53.809742 30128 query_context.cpp:610] Retrying ReportExecStatus: No more data to read.

W1212 13:05:56.643096 30058 pipeline_driver_executor.cpp:161] [Driver] Process error, query_id=5ded4d98-98a6-11ee-a80d-00163e354101, instance_id=5ded4d98-98a6-11ee-a80d-00163e354102, status=Memory limit exceeded: Memory of Query5ded4d98-98a6-11ee-a80d-00163e354101 exceed limit. Pipeline Backend: 172.16.132.124, fragment: 5ded4d98-98a6-11ee-a80d-00163e354102 Used: 4310842632, Limit: 4294967296. Mem usage has exceed the limit of single query, You can change the limit by set session variable query_mem_limit.

W1212 13:05:56.643121 30098 file_reader.cpp:527] FileReader::get_next failed. reason = Memory limit exceeded: Memory of Query5ded4d98-98a6-11ee-a80d-00163e354101 exceed limit. read and decompress page Used: 4312955400, Limit: 4294967296. Mem usage has exceed the limit of single query, You can change the limit by set session variable query_mem_limit.

image
image762×256 16.2 KB

#2

com.starrocks.common.UserException: Memory of Query5ded4d98-98a6-11ee-a80d-00163e354101 exceed limit. Pipeline Backend: 172.16.132.124, fragment: 5ded4d98-98a6-11ee-a80d-00163e354102 Used: 4310842632, Limit: 4294967296. Mem usage has exceed the limit of single query, You can change the limit by set session variable query_mem_limit.

#4

CREATE MATERIALIZED VIEW crawlerdb_3m
COMMENT “mv test”
PARTITION BY (date_trunc(‘day’, day))
DISTRIBUTED BY HASH(name) BUCKETS 2
– refresh_moment refresh_moment IMMEDIATE
– refresh_scheme REFRESH ASYNC START(“2023-07-01 10:00:00”) EVERY(INTERVAL 1 HOUR)
PROPERTIES (
“replication_num” = “1”,
“partition_ttl” = “3 MONTH”
)
AS
SELECT home_url, url, wiki, rating, channel, channeltype, episode, whole, episode_num, type, down, tag, up, specialtag,
index, play_times, commentnum, name, total_play_times_zheng_pian, play_times_zheng_pian, desc, updatetext, duration, total_play_times, home_name,
home_id, release_time, timestamp, releaser, isvip, total_episode_num, video_type, total_play_times_hua_xu,
dramatype, tv_id, ismemberfree, publish_time, isoffline,
indextext, ori_name, realchanneltype, area, ori_home_id, cover_url, heat, total_heat, str2date(day, ‘%Y-%m-%d’) AS day
FROM hive.es_parquet.crawlerdb where day >=‘2023-09-01’ ;

#5

您好，集群be内存资源充足的话可以调高下query_mem_limit参数。set global query_mem_limit = xxx;

#6

我看 文档 query_mem_limit = 0 是不限制， 还需要再单独配置么

#7

query_mem_limit=0的时候会按照exec_mem_limit来计算的

image
image1616×778 112 KB

#8

谢谢老师

#9

请问这个截图的文档有链接地址么， 我没有找到， 谢谢

#10
#11

请问这个变量可以配置在be.conf 里么

#12

这个是系统参数哈，set global是会持久化到fe的元数据里的。

京ICP备2022026421号-1