【详述】主键模型关联表update,报错Memory of process exceed limit.
【背景】
BE物理机内存128G,process 内存限制101G
主表 A 数据行:1.16亿,列数量:430,数据大小: 25.5G,主键模型表
子表 B 数据行:505万,列数量:15,数据大小: 3.7G,主键模型表
注意:子表B比这个量级还大的情况也都能更新成功(结构相同),就上面这一波数据稳定复现
关联表做update时,空闲内存有70G左右,执行sql后,报错如下:
2023-07-07 16:39:51,903 WARN (thrift-server-pool-64798|69721) [Coordinator.updateStatus():1291] one instance report fail throw updateStatus(), need cancel. job id: 253399, query id: 44a4142f-1ca1-11ee-a24e-02420b962997, instance id: 44a4142f-1ca1-11ee-a24e-02420b96299b
2023-07-07 16:39:51,904 WARN (starrocks-mysql-nio-pool-413|69351) [StmtExecutor.handleDMLStmt():1547] insert failed: Memory of process exceed limit. try consume:16777216 Used: 109168586256, Limit: 109182332436. Mem usage has exceed the limit of BE
2023-07-07 16:39:51,904 WARN (starrocks-mysql-nio-pool-413|69351) [StmtExecutor.handleDMLStmt():1655] failed to handle stmt [update cdp_tags_union set uniquecode = ‘f978e94d0d3b4420b8ca699ba47d016f’, tag_array = cdp_tags_union_tmp.tag_array, tag_bitmap = cdp_tags_union_tmp.tag_bitmap,tag_52573 = cdp_tags_union_tmp.tag_52573,tag_52567 = cdp_tags_union_tmp.tag_52567,tag_52570 = cdp_tags_union_tmp.tag_52570,tag_52574 = cdp_tags_union_tmp.tag_52574,tag_52572 = cdp_tags_union_tmp.tag_52572,tag_52569 = cdp_tags_union_tmp.tag_52569,tag_52566 = cdp_tags_union_tmp.tag_52566,tag_52571 = cdp_tags_union_tmp.tag_52571,tag_52575 = cdp_tags_union_tmp.tag_52575,tag_52568 = cdp_tags_union_tmp.tag_52568 from cdp_tags_union_tmp where cdp_tags_union.superid = cdp_tags_union_tmp.superid;] label: update_44a4142f-1ca1-11ee-a24e-02420b962997
com.starrocks.common.DdlException: Memory of process exceed limit. try consume:16777216 Used: 109168586256, Limit: 109182332436. Mem usage has exceed the limit of BE
at com.starrocks.common.ErrorReport.reportDdlException(ErrorReport.java:95) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleDMLStmt(StmtExecutor.java:1548) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleDMLStmtWithProfile(StmtExecutor.java:1340) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:554) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:349) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:463) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:729) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_372]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_372]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_372]
2023-07-07 16:39:51,907 WARN (starrocks-mysql-nio-pool-413|69351) [StmtExecutor.handleDMLStmtWithProfile():1346] DML statement(update cdp_tags_union set uniquecode = ‘f978e94d0d3b4420b8ca699ba47d016f’, tag_array = cdp_tags_union_tmp.tag_array, tag_bitmap = cdp_tags_union_tmp.tag_bitmap,tag_52573 = cdp_tags_union_tmp.tag_52573,tag_52567 = cdp_tags_union_tmp.tag_52567,tag_52570 = cdp_tags_union_tmp.tag_52570,tag_52574 = cdp_tags_union_tmp.tag_52574,tag_52572 = cdp_tags_union_tmp.tag_52572,tag_52569 = cdp_tags_union_tmp.tag_52569,tag_52566 = cdp_tags_union_tmp.tag_52566,tag_52571 = cdp_tags_union_tmp.tag_52571,tag_52575 = cdp_tags_union_tmp.tag_52575,tag_52568 = cdp_tags_union_tmp.tag_52568 from cdp_tags_union_tmp where cdp_tags_union.superid = cdp_tags_union_tmp.superid;) process failed.
com.starrocks.common.UserException: Memory of process exceed limit. try consume:16777216 Used: 109168586256, Limit: 109182332436. Mem usage has exceed the limit of BE
at com.starrocks.qe.StmtExecutor.handleDMLStmt(StmtExecutor.java:1699) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleDMLStmtWithProfile(StmtExecutor.java:1340) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:554) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:349) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:463) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:729) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_372]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_372]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_372]
sql语句如下:
update A
set uniquecode = 'cddf82c74915498f9e3157678166304f',
column1 = B.column1 ,
column2 = B.column2 ,
column3 = B.column3 ,
column4 = B.column4 ,
column5 = B.column5 ,
column6 = B.column6 ,
column7 = B.column7 ,
column8 = B.column8 ,
column9 = B.column9 ,
column10 = B.column10,
column11 = B.column11,
column12 = B.column12,
from B
where A.x = B.x
【业务影响】更新失败
【StarRocks版本】3.0.3-fe5e3a1
【集群规模】例如:1fe + 8be
【机器信息】 32C/128G/千兆 * 9
【联系方式】gaocj314@163.com
请问:为何此操作比较耗费内存,有什么方法可以避免吗。