fe OOM

starrocks批量使用insert into导入数据时
starrocks version 2.5.1
fe oom
fe节点日志:
Exception in thread “thrift-server-pool-9782” java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:410)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:254)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27)
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:311)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Exception in thread “thrift-server-pool-14430” java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:705)
at java.util.HashMap.putVal(HashMap.java:630)
at java.util.HashMap.put(HashMap.java:613)
at com.starrocks.thrift.TReportRequest$TReportRequestStandardScheme.read(TReportRequest.java:1221)
at com.starrocks.thrift.TReportRequest$TReportRequestStandardScheme.read(TReportRequest.java:1147)
at com.starrocks.thrift.TReportRequest.read(TReportRequest.java:1008)
at com.starrocks.thrift.FrontendService$report_args$report_argsStandardScheme.read(FrontendService.java:14170)
at com.starrocks.thrift.FrontendService$report_args$report_argsStandardScheme.read(FrontendService.java:14155)
at com.starrocks.thrift.FrontendService$report_args.read(FrontendService.java:14102)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:23)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:311)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Exception in thread “thrift-server-pool-14438” java.lang.OutOfMemoryError: Java heap space
Exception in thread “thrift-server-pool-14459” java.lang.OutOfMemoryError: Java heap space
Exception in thread “thrift-server-pool-14460” java.lang.OutOfMemoryError: Java heap space
Exception in thread “thrift-server-pool-14461” java.lang.OutOfMemoryError: Java heap space
Exception in thread “thrift-server-pool-14475” java.lang.OutOfMemoryError: Java heap space

fe重启后 fe.out日志如下
Mar 15, 2023 2:47:51 AM com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncBulkCompleter accept
WARNING: Exception thrown during asynchronous load
java.util.concurrent.CompletionException: com.starrocks.sql.analyzer.SemanticException: get_applied_rowsets(version 21997) failed tablet:11063 #version:174 [22321 22486.1@173 22486.1] #pending:0
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: com.starrocks.sql.analyzer.SemanticException: get_applied_rowsets(version 21997) failed tablet:11063 #version:174 [22321 22486.1@173 22486.1] #pending:0
at com.starrocks.statistic.StatisticExecutor.executeDQL(StatisticExecutor.java:243)
at com.starrocks.statistic.StatisticExecutor.queryStatisticSync(StatisticExecutor.java:83)
at com.starrocks.sql.optimizer.statistics.ColumnBasicStatsCacheLoader.queryStatisticsData(ColumnBasicStatsCacheLoader.java:111)
at com.starrocks.sql.optimizer.statistics.ColumnBasicStatsCacheLoader.lambda$asyncLoadAll$1(ColumnBasicStatsCacheLoader.java:77)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
… 5 more

Mar 15, 2023 2:47:51 AM com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncBulkCompleter accept
WARNING: Exception thrown during asynchronous load
java.util.concurrent.CompletionException: com.starrocks.sql.analyzer.SemanticException: get_applied_rowsets(version 22202) failed tablet:11063 #version:138 [22356 22486.1@137 22486.1] #pending:0
at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1606)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: com.starrocks.sql.analyzer.SemanticException: get_applied_rowsets(version 22202) failed tablet:11063 #version:138 [22356 22486.1@137 22486.1] #pending:0
at com.starrocks.statistic.StatisticExecutor.executeDQL(StatisticExecutor.java:243)
at com.starrocks.statistic.StatisticExecutor.queryStatisticSync(StatisticExecutor.java:83)
at com.starrocks.sql.optimizer.statistics.ColumnBasicStatsCacheLoader.queryStatisticsData(ColumnBasicStatsCacheLoader.java:111)
at com.starrocks.sql.optimizer.statistics.ColumnBasicStatsCacheLoader.lambda$asyncLoadAll$1(ColumnBasicStatsCacheLoader.java:77)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
… 5 more

重启后fe可以正常连接,但是过段时间后,会报OOM错误

jvm 的Xmx设置的是多大,集群什么配置

StarRocks 2.5.2修复过1个由Insert插入导致的内存泄露问题,可以看下监控图FE是不是一直在内存泄露

默认值,没有修改,-Xmx8192m, 集群内存200多G,1个fe, 四个be

但是重启后,目前没有insert 插入的动作了,还是回OOM, fe.out里面有一个异常

现在把sr里面所有的表都删除掉后,还是回GC, 重启一样会报同样的错误

是FE leader OOM,还是Follower?

只装了一个fe, fe leader

@trueeyu


这个报错是因为什么呢?现在我把所有表删除了,还是会OOM

你能把jvm改成16G重启下不,8G有点小 java heap space

好的,我调整下,请问下,图片中那个报错是为啥?还有我现在把所有表都删除了,为什么还需要这么大内存呢?谢谢解答

改成16G还启动不了么。看下fe/meta文件夹大么

我这边重新装了2.5.3的,现在依旧会存在OOM的情况,我调成16G后,在数据导入的会OOM, 然后重启依旧会OOM meta文件夹下面,bdp1.2G, image 902M

使用jstat -gcutil 340932 5000 指令查看发现,FGC和YGC频率挺高的

我这边有dump文件,不过是上一次8G内存的,如果需要,可以发你

fe leader 节点自动退出,退出前执行了jmap -histo:live pid fe.out日志如下:
fe_out.txt (329.4 KB)

fe jvm内存设置为32G