【详述】日志一直报Database 5307842 has been forcibly closed in order to apply a replicated remove operation. This Database and all associated Cursors must be closed. All associated Transactions must be aborted
【业务影响】
【StarRocks版本】2.3.4
【集群规模】例如：2fe+6be（fe与be混部）
【机器信息】16核 32线程
com.sleepycat.je.rep.DatabasePreemptedException: (JE 7.3.7) (JE 7.3.7) Database 5257842 has been forcibly closed in order to apply a replicated remove operation. This Database and all associated Cursors must be closed. All associated Transactions must be aborted.
at com.sleepycat.je.rep.DatabasePreemptedException.wrapSelf(DatabasePreemptedException.java:113) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Database.checkOpen(Database.java:2274) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Database.getDatabaseName(Database.java:2046) ~[je-7.3.7.jar:7.3.7]
at com.starrocks.journal.bdbje.BDBEnvironment.openDatabase(BDBEnvironment.java:286) [starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBJEJournal.getMaxJournalId(BDBJEJournal.java:223) [starrocks-fe.jar:?]
at com.starrocks.persist.EditLog.getMaxJournalId(EditLog.java:105) [starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.getMaxJournalId(GlobalStateMgr.java:2437) [starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.replayJournal(GlobalStateMgr.java:1705) [starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr$4.runOneCycle(GlobalStateMgr.java:1518) [starrocks-fe.jar:?]
at com.starrocks.common.util.Daemon.run(Daemon.java:115) [starrocks-fe.jar:?]
Caused by: com.sleepycat.je.rep.DatabasePreemptedException: (JE 7.3.7) Database 5257842 has been forcibly closed in order to apply a replicated remove operation. This Database and all associated Cursors must be closed. All associated Transactions must be aborted.
at com.sleepycat.je.rep.impl.RepImpl.createDatabasePreemptedException(RepImpl.java:2008) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.createDatabasePreemptedException(RepImpl.java:143) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Database.setPreempted(Database.java:469) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.DbInternal.setPreempted(DbInternal.java:58) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.dbi.DbTree.lockNameLN(DbTree.java:972) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.dbi.DbTree.doRemoveDb(DbTree.java:1172) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.dbi.DbTree.removeReplicaDb(DbTree.java:1239) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.node.Replay.applyNameLN(Replay.java:872) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:598) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.node.Replica$ReplayThread.run(Replica.java:1213) ~[je-7.3.7.jar:7.3.7]
2022-12-15 09:11:02,274 WARN (starrocks-mysql-nio-pool-1412|1903) [ReadListener.lambda$handleEvent$0():63] Exception happened in one session(com.starrocks.mysql.nio.NConnectContext@2f629e3).
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_181]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_181]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_181]
at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_181]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_181]
at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:289) ~[xnio-nio-3.7.9.Final.jar:3.7.9.Final]
at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127) ~[xnio-api-3.7.9.Final.jar:3.7.9.Final]
at org.xnio.channels.Channels.readBlocking(Channels.java:294) ~[xnio-api-3.7.9.Final.jar:3.7.9.Final]
at com.starrocks.mysql.nio.NMysqlChannel.readAll(NMysqlChannel.java:63) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.MysqlChannel.fetchOnePacket(MysqlChannel.java:156) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:673) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2022-12-15 09:11:02,274 WARN (starrocks-mysql-nio-pool-1415|1906) [ReadListener.lambda$handleEvent$0():63] Exception happened in one session(com.starrocks.mysql.nio.NConnectContext@5b26b45a).
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.8.0_181]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.8.0_181]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.8.0_181]
at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[?:1.8.0_181]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[?:1.8.0_181]
at org.xnio.nio.NioSocketConduit.read(NioSocketConduit.java:289) ~[xnio-nio-3.7.9.Final.jar:3.7.9.Final]
at org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127) ~[xnio-api-3.7.9.Final.jar:3.7.9.Final]
at org.xnio.channels.Channels.readBlocking(Channels.java:294) ~[xnio-api-3.7.9.Final.jar:3.7.9.Final]
at com.starrocks.mysql.nio.NMysqlChannel.readAll(NMysqlChannel.java:63) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.MysqlChannel.fetchOnePacket(MysqlChannel.java:156) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:673) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
2022-12-15 10:51:54,960 WARN (replayer|76) [BDBEnvironment.openDatabase():310] get exception when try to close previously opened bdb database. ignore it
com.sleepycat.je.rep.DatabasePreemptedException: (JE 7.3.7) (JE 7.3.7) Database 5307842 has been forcibly closed in order to apply a replicated remove operation. This Database and all associated Cursors must be closed. All associated Transactions must be aborted.
at com.sleepycat.je.rep.DatabasePreemptedException.wrapSelf(DatabasePreemptedException.java:113) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Database.checkOpen(Database.java:2274) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Database.getDatabaseName(Database.java:2046) ~[je-7.3.7.jar:7.3.7]
at com.starrocks.journal.bdbje.BDBEnvironment.openDatabase(BDBEnvironment.java:286) [starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBJEJournal.getMaxJournalId(BDBJEJournal.java:223) [starrocks-fe.jar:?]
at com.starrocks.persist.EditLog.getMaxJournalId(EditLog.java:105) [starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.getMaxJournalId(GlobalStateMgr.java:2437) [starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.replayJournal(GlobalStateMgr.java:1705) [starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr$4.runOneCycle(GlobalStateMgr.java:1518) [starrocks-fe.jar:?]
at com.starrocks.common.util.Daemon.run(Daemon.java:115) [starrocks-fe.jar:?]
Caused by: com.sleepycat.je.rep.DatabasePreemptedException: (JE 7.3.7) Database 5307842 has been forcibly closed in order to apply a replicated remove operation. This Database and all associated Cursors must be closed. All associated Transactions must be aborted.
at com.sleepycat.je.rep.impl.RepImpl.createDatabasePreemptedException(RepImpl.java:2008) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.createDatabasePreemptedException(RepImpl.java:143) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Database.setPreempted(Database.java:469) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.DbInternal.setPreempted(DbInternal.java:58) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.dbi.DbTree.lockNameLN(DbTree.java:972) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.dbi.DbTree.doRemoveDb(DbTree.java:1172) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.dbi.DbTree.removeReplicaDb(DbTree.java:1239) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.node.Replay.applyNameLN(Replay.java:872) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.node.Replay.replayEntry(Replay.java:598) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.node.Replica$ReplayThread.run(Replica.java:1213) ~[je-7.3.7.jar:7.3.7]
fe日志当中如果后面有 ignore it 字样，是可以忽略无需处理的日志信息