insert into select报Mem usage has exceed the limit of single query, You can change the limit by set session variable exec_mem_limit or query_mem_limit.

【详述】操作insert into select时报
one instance report fail errorCode MEM_LIMIT_EXCEEDED Memory of Fragment 202ee318-8100-11ed-8600-525400ed2b7d exceed limit. read and decompress page Used: 12886247496, Limit: 12884901888. Mem usage has exceed the limit of single query, You can change the limit by set session variable exec_mem_limit or query_mem_limit
表数据量只有500多M,正常scan/fliter/load,任何一个fragement都不应该触及12G。
【背景】做过哪些操作?
【业务影响】
【StarRocks版本】例如:2.3.3
【集群规模】例如:3fe(1 follower+2observer)+5be(fe与be分部)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【表模型】例如:更新模型
fe.log日志

    2022-12-21 15:21:58,216 WARN (thrift-server-pool-99697|143783) [Coordinator.updateFragmentExecStatus():1645] one instance report fail errorCode MEM_LIMIT_EXCEEDED Memory of Fragment 202ee318-8100-11ed-8600-525400ed2b7d exceed limit. read and decompress page Used: 12886247496, Limit: 12884901888. Mem usage has exceed the limit of single query, You can change the limit by set session variable exec_mem_limit or query_mem_limit., query_id=202ee318-8100-11ed-8600-525400ed2b7b instance_id=202ee318-8100-11ed-8600-525400ed2b7d
    2022-12-21 15:21:58,216 WARN (thrift-server-pool-99697|143783) [Coordinator.updateStatus():870] one instance report fail throw updateStatus(), need cancel. job id: -1, query id: 202ee318-8100-11ed-8600-525400ed2b7b, instance id: 202ee318-8100-11ed-8600-525400ed2b7d
    2022-12-21 15:21:58,216 INFO (thrift-server-pool-99697|143783) [Coordinator.cancelInternal():966] unfinished instance: 202ee318-8100-11ed-8600-525400ed2b7d
    2022-12-21 15:21:58,216 WARN (starrocks-mysql-nio-pool-10510|143932) [StmtExecutor.handleDMLStmt():1153] insert failed: Memory of Fragment 202ee318-8100-11ed-8600-525400ed2b7d exceed limit. read and decompress page Used: 12886247496, Limit: 12884901888. Mem usage has exceed the limit of single query, You can change the limit by set session variable exec_mem_limit or query_mem_limit.
    2022-12-21 15:21:58,216 WARN (starrocks-mysql-nio-pool-10510|143932) [StmtExecutor.handleDMLStmt():1229] handle insert stmt fail: insert_202ee318-8100-11ed-8600-525400ed2b7b
    com.starrocks.common.DdlException: Memory of Fragment 202ee318-8100-11ed-8600-525400ed2b7d exceed limit. read and decompress page Used: 12886247496, Limit: 12884901888. Mem usage has exceed the limit of single query, You can change the limit by set session variable exec_mem_limit or query_mem_limit.
            at com.starrocks.common.ErrorReport.reportDdlException(ErrorReport.java:80) ~[starrocks-fe.jar:?]
            at com.starrocks.qe.StmtExecutor.handleDMLStmt(StmtExecutor.java:1154) ~[starrocks-fe.jar:?]
            at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:430) ~[starrocks-fe.jar:?]
            at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:322) ~[starrocks-fe.jar:?]
            at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:440) ~[starrocks-fe.jar:?]
            at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:676) ~[starrocks-fe.jar:?]
            at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]
            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]
    2022-12-21 15:21:58,287 INFO (starrocks-mysql-nio-pool-10510|143932) [QeProcessorImpl.unregisterQuery():79] deregister query id 202ee318-8100-11ed-8600-525400ed2b7b

fe.audit.log

2022-12-21 15:21:58,294 [slow_query] |Client=10.10.30.7:40570|User=default_cluster:u_ci|AuthorizedUser='default_cluster:u_ci'@'%'|ResourceGroup=default_wg|Catalog=default_catalog|Db=default_cluster:labelx|State=ERR|ErrorCod
e=MEM_LIMIT_EXCEEDED|Time=12459|ScanBytes=0|ScanRows=0|ReturnRows=0|CpuCostNs=0|MemCostBytes=0|StmtId=6622491|QueryId=202ee318-8100-11ed-8600-525400ed2b7b|IsQuery=false|feIp=10.10.20.5|Stmt=INSERT INTO user_coupon         s
elect user_id,trans_no, amount,  global_biz_id,1,verification, take_date, take_time,out_side_biz_id,dt from       user_coupon              where  deleted = 0  and datediff(CURDATE(), take_date)>= 7|Digest=|PlanCpuCost=2.090
6963E7|PlanMemCost=0.0

sql语句:

select user_id,trans_no, amount, global_biz_id,1,verification, take_date, take_time,out_side_biz_id,dt from user_coupon where deleted = 0 and datediff(CURDATE(), take_date)>= 7;

建表语句:

CREATE TABLE `user_coupon` (
  `user_id` bigint(20) NOT NULL ,
  `trans_no` varchar(50) NOT NULL ,
  `amount` decimal64(10, 2) NULL ,
  `global_biz_id` varchar(50) NULL ,
  `deleted` tinyint(4) NULL DEFAULT "0" ,
  `verification` tinyint(4) NULL DEFAULT "0" ,
  `take_date` date NULL ,
  `take_time` datetime NULL ,
  `out_side_biz_id` varchar(50) NULL ,
  `dt` date NOT NULL 
) ENGINE=OLAP 
UNIQUE KEY(`user_id`, `trans_no`)
DISTRIBUTED BY HASH(`user_id`) BUCKETS 3 
PROPERTIES (
"replication_num" = "3",
"in_memory" = "false",
"storage_format" = "DEFAULT",
"enable_persistent_index" = "false"
);

表数据量:

使用的starrocks具体版本是?

版本是2.3.3

麻烦吧这个内存调大点跑通一次,然后下载profile看看呢,另外发下explain sql和explain costs sql的结果,谢谢

后来调大跑了是没问题,但是这个表业务使用时会truncate,导致数据量和出现问题时不一致。后续这里在出问题后,我把相关信息上传上来。

这个问题,后来又出现过吗?一起看下这个问题?