ERROR Runtime error: failed memory alloc in constructor

为了更快的定位您的问题,请提供以下信息,谢谢
【详述】
ISNERT INTO的时候出现ERROR Runtime error: failed memory alloc in constructor
【背景】
多表关联导入Bitmap出现failed memory alloc in constructor,优化SQL之后改写成Bitmap还是不行
【业务影响】
为上线,测试环境中
【StarRocks版本】
2.3.13 974398d
【集群规模】例如:3fe(1 follower+2observer)+ 7be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】xujianjun159@gmail.com
【附件】
INSERT INTO tag_user_uploaded
SELECT *
FROM (
WITH init_event AS (
SELECT bitmap_union(to_bitmap(mapid)) users, enddate
FROM xxxx
WHERE enddate BETWEEN ‘2023-07-10’ AND ‘2023-07-17’
AND product_id = 18
AND plat_name = 17
AND page_id = 227
AND BITMAP_CONTAINS((
SELECT BITMAP_UNION(users)
FROM tag_user_bitmap_string
WHERE id = 101622
AND tag_id = ‘fvio’
AND platform = ‘ALL’
AND tag_value IN (‘1’)
), mapid)
group by enddate
),
retention_event AS (
SELECT bitmap_union(to_bitmap(mapid)) users, enddate
FROM xxxx
WHERE enddate BETWEEN ‘2023-07-10’ AND ‘2023-07-24’
AND product_id = 18
AND plat_name = 17
AND page_id = 121014
group by enddate
),
retention_table AS (
SELECT bitmap_and(a.users, b.users) users, DATEDIFF(b.enddate, a.enddate) AS date_diff, a.enddate AS enddate
, ‘’ AS tag_value
FROM init_event a
INNER JOIN retention_event b ON a.enddate <= b.enddate
WHERE DATEDIFF(b.enddate, a.enddate) >= 0
AND DATEDIFF(b.enddate, a.enddate) <= 7
)
SELECT ‘TEST_0727’, 3
, (
SELECT users
FROM retention_table
WHERE date_diff = 1
AND enddate = ‘2023-07-12’
)
) t

  • 慢查询:
  • 查询报错:
    ERROR Runtime error: failed memory alloc in constructor

这个问题也有其它用户报过,这个错是roaring bitmap报出来的,报的错不太好,需要优化下报错信息。同时让把exec_mem_limit或是query_mem_limit调大试试,看下能跑过吗。然后发我们一个explain costs。同时看下BE内存占用高吗,如果内存高的话,需要优化下SQL。当前我们也在优化这块,需要看下你的场景,当前是那里吃内存比较多。

其它: bitmap_union(to_bitmap(xxx)) 建议改成bitmap_agg(xxx) 可以提升性能。

实际上我不清楚bitmap_union(to_bitmap(xxx))和bitmap_agg(xxx) 两个函数在使用内存上面的差异,但是实际测试发现bitmap_agg比bitmap_union(to_bitmap(xxx))差很多,实际上我改写这个SQL之后就可以执行,改写的SQL我感觉比这个还复杂,有Cross Join都能成功执行

SQL改写成这样可以成功执行:
INSERT INTO tag_user_uploaded
SELECT *
FROM (
WITH init_event AS (
SELECT bitmap_union(to_bitmap(mapid)) AS users, enddate
FROM flow_plat_page_detail_retention
WHERE enddate BETWEEN ‘2023-07-10’ AND ‘2023-07-17’
AND product_id = 18
AND plat_name = 17
AND page_id = 227
GROUP BY enddate
),
filter_condition AS (
SELECT BITMAP_UNION(users) AS users
FROM tag_user_bitmap_string
WHERE id = xx
AND tag_id = ‘xxx’
AND platform = ‘ALL’
AND tag_value IN (‘1’)
),
init_event_result AS (
SELECT bitmap_and(a.users, b.users) users, a.enddate as enddate
FROM init_event a
CROSS JOIN filter_condition b
),
retention_event AS (
SELECT bitmap_union(to_bitmap(mapid)) AS users, enddate
FROM flow_plat_page_detail_retention
WHERE enddate BETWEEN ‘2023-07-10’ AND ‘2023-07-24’
AND product_id = xx
AND plat_name = xx
AND page_id = xx
GROUP BY enddate
),
retention_table AS (
SELECT bitmap_and(a.users, b.users) AS users
, DATEDIFF(b.enddate, a.enddate) AS date_diff, a.enddate AS enddate
, ‘’ AS tag_value
FROM init_event_result a
INNER JOIN retention_event b ON a.enddate <= b.enddate
WHERE DATEDIFF(b.enddate, a.enddate) >= 0
AND DATEDIFF(b.enddate, a.enddate) <= 7
)
SELECT ‘JJ_TEST_0727’, 3
, (
SELECT users
FROM retention_table
WHERE date_diff = 1
AND enddate = ‘2023-07-12’
)
) t

当前query_mem_limit和exec_mem_limit 分别设置 的是多少,可以加个微信聊下?最近我在优化Bitmap这块的内存消耗和优化性能

请问解决了没有?

你用的是什么版本?

2.5.13

2.5.19解决了,升级下吧

好的 谢谢