【详述】问题详细描述
【背景】做过哪些操作？
【业务影响】
【StarRocks版本】2.3.0
【集群规模】3fe+4be
【机器信息】16h32g
【联系方式】137605752@qq.com
【附件】
2023-02-23 10:27:16,252 ERROR (starrocks-mysql-nio-pool-29|427) [BDBEnvironment.getDatabaseNames():395] bdb environment failure exception.
com.sleepycat.je.EnvironmentFailureException: (JE 7.3.7) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 7.3.7) 10.150.9.31_9010_1672122689706(12):/opt/starrocks/fe/meta/bdb Uncaught exception in feeder thread:Thread[Feeder Output for 10.150.9.33_9010_1677079514722,5,main] UNCAUGHT_EXCEPTION: Uncaught Exception in internal thread, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:228) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1766) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1775) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Environment.checkOpen(Environment.java:2473) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Environment.getDatabaseNames(Environment.java:2245) ~[je-7.3.7.jar:7.3.7]
at com.starrocks.journal.bdbje.BDBEnvironment.getDatabaseNames(BDBEnvironment.java:380) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBJEJournal.getMaxJournalId(BDBJEJournal.java:212) ~[starrocks-fe.jar:?]
at com.starrocks.persist.EditLog.getMaxJournalId(EditLog.java:105) ~[starrocks-fe.jar:?]
at com.starrocks.common.proc.FrontendsProcNode.getFrontendsInfo(FrontendsProcNode.java:110) ~[starrocks-fe.jar:?]
at com.starrocks.common.proc.FrontendsProcNode.fetchResult(FrontendsProcNode.java:64) ~[starrocks-fe.jar:?]
at com.starrocks.analysis.ShowProcStmt.getMetaData(ShowProcStmt.java:76) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ShowExecutor.handleShowProc(ShowExecutor.java:443) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ShowExecutor.execute(ShowExecutor.java:177) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleShow(StmtExecutor.java:912) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:451) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:283) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:401) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:637) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_301]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_301]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_301]
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 7.3.7) 10.150.9.31_9010_1672122689706(12):/opt/starrocks/fe/meta/bdb Uncaught exception in feeder thread:Thread[Feeder Output for 10.150.9.33_9010_1677079514722,5,main] UNCAUGHT_EXCEPTION: Uncaught Exception in internal thread, unable to continue. Environment is invalid and must be closed.
at com.sleepycat.je.EnvironmentFailureException.promote(EnvironmentFailureException.java:324) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.node.Feeder$IOThreadsHandler.uncaughtException(Feeder.java:1538) ~[je-7.3.7.jar:7.3.7]
at java.lang.Thread.dispatchUncaughtException(Thread.java:1959) ~[?:1.8.0_301]
Caused by: java.lang.OutOfMemoryError: Java heap space
OOM了，看看监控是只有follower内存涨吗？
在follower上执行一下 jmap -histo:live pid发一下。
我通过升级2.4.3没出现这个问题了
