FE挂死，帮忙看下具体原因

dzk2043 · 2025年03月28日 02:24

【详述】FE挂死
【背景】平常会有大批量的查询和插入操作
【业务影响】
【是否存算分离】
【StarRocks版本】2.5.6-a193ae0
【集群规模】3fe（1 follower+2observer）+3be（fe与be混部）
【机器信息】32C/64G
【联系方式】406166876@qq.com
【报错日志】

2025-03-28 00:15:27,358 ERROR (JournalWriter|82) [BDBJEJournal.batchWriteCommit():407] failed to commit journal after retried 1 times! txn[] db[CloseSafeDatabase{db=369915905}]
com.sleepycat.je.rep.InsufficientAcksException: (JE 7.3.7) Transaction: -384244135 VLSN: 746,005,380, initiated at: 00:15:17. Insufficient acks for policy:SIMPLE_MAJORITY. Need replica acks: 1. Missing replica acks: 1. Timeout: 10000ms. FeederState=192.168.0.23_9010_1658221874592(1)[MASTER]
Current feeds:
192.168.0.33_9010_1741855979966: feederVLSN=746,005,381 replicaTxnEndVLSN=746,005,378
192.168.0.68_9010_1741855982823: feederVLSN=746,005,381 replicaTxnEndVLSN=746,005,378

at com.sleepycat.je.rep.impl.node.DurabilityQuorum.ensureSufficientAcks(DurabilityQuorum.java:205) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.stream.FeederTxns.awaitReplicaAcks(FeederTxns.java:189) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.postLogCommitHookInternal(RepImpl.java:1426) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.postLogCommitHook(RepImpl.java:1385) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.txn.MasterTxn.postLogCommitHook(MasterTxn.java:226) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.commit(Txn.java:772) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.commit(Txn.java:625) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Transaction.commit(Transaction.java:337) ~[je-7.3.7.jar:7.3.7]
at com.starrocks.journal.bdbje.BDBJEJournal.batchWriteCommit(BDBJEJournal.java:402) [starrocks-fe.jar:?]
at com.starrocks.journal.JournalWriter.writeOneBatch(JournalWriter.java:114) [starrocks-fe.jar:?]
at com.starrocks.journal.JournalWriter$1.runOneCycle(JournalWriter.java:74) [starrocks-fe.jar:?]
at com.starrocks.common.util.Daemon.run(Daemon.java:115) [starrocks-fe.jar:?]

2025-03-28 00:15:32,358 WARN (JournalWriter|82) [BDBJEJournal.rebuildCurrentTransaction():427] transaction is invalid, rebuild the txn with 1 kvs
2025-03-28 00:15:42,370 ERROR (JournalWriter|82) [BDBJEJournal.batchWriteCommit():407] failed to commit journal after retried 2 times! txn[] db[CloseSafeDatabase{db=369915905}]
com.sleepycat.je.rep.InsufficientAcksException: (JE 7.3.7) Transaction: -384244138 VLSN: 746,005,382, initiated at: 00:15:32. Insufficient acks for policy:SIMPLE_MAJORITY. Need replica acks: 1. Missing replica acks: 1. Timeout: 10000ms. FeederState=192.168.0.23_9010_1658221874592(1)[MASTER]
Current feeds:
192.168.0.33_9010_1741855979966: feederVLSN=746,005,383 replicaTxnEndVLSN=746,005,378
192.168.0.68_9010_1741855982823: feederVLSN=746,005,383 replicaTxnEndVLSN=746,005,378

at com.sleepycat.je.rep.impl.node.DurabilityQuorum.ensureSufficientAcks(DurabilityQuorum.java:205) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.stream.FeederTxns.awaitReplicaAcks(FeederTxns.java:189) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.postLogCommitHookInternal(RepImpl.java:1426) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.postLogCommitHook(RepImpl.java:1385) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.txn.MasterTxn.postLogCommitHook(MasterTxn.java:226) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.commit(Txn.java:772) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.commit(Txn.java:625) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Transaction.commit(Transaction.java:337) ~[je-7.3.7.jar:7.3.7]
at com.starrocks.journal.bdbje.BDBJEJournal.batchWriteCommit(BDBJEJournal.java:402) [starrocks-fe.jar:?]
at com.starrocks.journal.JournalWriter.writeOneBatch(JournalWriter.java:114) [starrocks-fe.jar:?]
at com.starrocks.journal.JournalWriter$1.runOneCycle(JournalWriter.java:74) [starrocks-fe.jar:?]
at com.starrocks.common.util.Daemon.run(Daemon.java:115) [starrocks-fe.jar:?]

2025-03-28 00:15:47,371 WARN (JournalWriter|82) [BDBJEJournal.rebuildCurrentTransaction():427] transaction is invalid, rebuild the txn with 1 kvs
2025-03-28 00:15:57,394 ERROR (JournalWriter|82) [BDBJEJournal.batchWriteCommit():407] failed to commit journal after retried 3 times! txn[] db[CloseSafeDatabase{db=369915905}]
com.sleepycat.je.rep.InsufficientAcksException: (JE 7.3.7) Transaction: -384244141 VLSN: 746,005,384, initiated at: 00:15:47. Insufficient acks for policy:SIMPLE_MAJORITY. Need replica acks: 1. Missing replica acks: 1. Timeout: 10000ms. FeederState=192.168.0.23_9010_1658221874592(1)[MASTER]
Current feeds:
192.168.0.33_9010_1741855979966: feederVLSN=746,005,387 replicaTxnEndVLSN=746,005,378
192.168.0.68_9010_1741855982823: feederVLSN=746,005,387 replicaTxnEndVLSN=746,005,378

at com.sleepycat.je.rep.impl.node.DurabilityQuorum.ensureSufficientAcks(DurabilityQuorum.java:205) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.stream.FeederTxns.awaitReplicaAcks(FeederTxns.java:189) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.postLogCommitHookInternal(RepImpl.java:1426) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.postLogCommitHook(RepImpl.java:1385) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.txn.MasterTxn.postLogCommitHook(MasterTxn.java:226) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.commit(Txn.java:772) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.commit(Txn.java:625) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Transaction.commit(Transaction.java:337) ~[je-7.3.7.jar:7.3.7]
at com.starrocks.journal.bdbje.BDBJEJournal.batchWriteCommit(BDBJEJournal.java:402) [starrocks-fe.jar:?]
at com.starrocks.journal.JournalWriter.writeOneBatch(JournalWriter.java:114) [starrocks-fe.jar:?]
at com.starrocks.journal.JournalWriter$1.runOneCycle(JournalWriter.java:74) [starrocks-fe.jar:?]
at com.starrocks.common.util.Daemon.run(Daemon.java:115) [starrocks-fe.jar:?]

2025-03-28 00:15:57,394 WARN (JournalWriter|82) [JournalWriter.writeOneBatch():120] failed to commit batch, will abort current 1 journals.
com.starrocks.journal.JournalException: failed to commit journal after retried 3 times! txn[] db[CloseSafeDatabase{db=369915905}]
at com.starrocks.journal.bdbje.BDBJEJournal.batchWriteCommit(BDBJEJournal.java:408) ~[starrocks-fe.jar:?]
at com.starrocks.journal.JournalWriter.writeOneBatch(JournalWriter.java:114) [starrocks-fe.jar:?]
at com.starrocks.journal.JournalWriter$1.runOneCycle(JournalWriter.java:74) [starrocks-fe.jar:?]
at com.starrocks.common.util.Daemon.run(Daemon.java:115) [starrocks-fe.jar:?]
Caused by: com.sleepycat.je.rep.InsufficientAcksException: (JE 7.3.7) Transaction: -384244141 VLSN: 746,005,384, initiated at: 00:15:47. Insufficient acks for policy:SIMPLE_MAJORITY. Need replica acks: 1. Missing replica acks: 1. Timeout: 10000ms. FeederState=192.168.0.23_9010_1658221874592(1)[MASTER]
Current feeds:
192.168.0.33_9010_1741855979966: feederVLSN=746,005,387 replicaTxnEndVLSN=746,005,378
192.168.0.68_9010_1741855982823: feederVLSN=746,005,387 replicaTxnEndVLSN=746,005,378

at com.sleepycat.je.rep.impl.node.DurabilityQuorum.ensureSufficientAcks(DurabilityQuorum.java:205) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.stream.FeederTxns.awaitReplicaAcks(FeederTxns.java:189) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.postLogCommitHookInternal(RepImpl.java:1426) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.postLogCommitHook(RepImpl.java:1385) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.txn.MasterTxn.postLogCommitHook(MasterTxn.java:226) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.commit(Txn.java:772) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.commit(Txn.java:625) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.Transaction.commit(Transaction.java:337) ~[je-7.3.7.jar:7.3.7]
at com.starrocks.journal.bdbje.BDBJEJournal.batchWriteCommit(BDBJEJournal.java:402) ~[starrocks-fe.jar:?]
... 3 more

2025-03-28 00:15:57,394 WARN (JournalWriter|82) [BDBJEJournal.batchWriteAbort():463] failed to abort transaction because no running transaction, will just ignore and return.
2025-03-28 00:15:57,394 ERROR (JournalWriter|82) [JournalWriter.abortJournalTask():163] failed to commit journal after retried 3 times! txn[] db[CloseSafeDatabase{db=369915905}]