【详述】无特别操作。
【背景】无特别操作。
【业务影响】
【StarRocks版本 2.5.4
【集群规模】例如:3fe + 3be
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息,请补充下您的联系方式,例如:社区群4-小李或者邮箱,谢谢
3节点FE服务,FE master节点,突然下线。从16点49开始出现报错:
2023-06-26 16:59:42,063 ERROR (heartbeat mgr|32) [Daemon.run():117] daemon thread got exception. name: heartbeat mgr
com.sleepycat.je.rep.InsufficientReplicasException: (JE 7.3.7) Commit policy: SIMPLE_MAJORITY required 1 replica. But none were active with this master.
at com.sleepycat.je.rep.impl.node.DurabilityQuorum.ensureReplicasForCommit(DurabilityQuorum.java:116) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.txnBeginHook(RepImpl.java:1161) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.txn.MasterTxn.txnBeginHook(MasterTxn.java:193) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.initTxn(Txn.java:377) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.<init>(Txn.java:286) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.<init>(Txn.java:265) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.txn.MasterTxn.<init>(MasterTxn.java:144) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.txn.MasterTxn$1.create(MasterTxn.java:115) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.txn.MasterTxn.create(MasterTxn.java:433) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.rep.impl.RepImpl.createRepUserTxn(RepImpl.java:1135) ~[je-7.3.7.jar:7.3.7]
at com.sleepycat.je.txn.Txn.createAutoTxn(Txn.java:332) ~[je-7.3.7.jar:7.3.7]
2023-06-26 16:59:42,197 ERROR (thrift-server-pool-2862|4178) [SRTThreadPoolServer$WorkerProcess.run():319] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed)
at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:60) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:311) [starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_291]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_291]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291]
Caused by: java.net.SocketException: Broken pipe (Write failed)
2023-06-26 17:00:53,279 ERROR (leaderCheckpointer|483) [Daemon.run():117] daemon thread got exception. name: leaderCheckpointer
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332) ~[?:1.8.0_291]
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) ~[?:1.8.0_291]
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448) ~[?:1.8.0_291]
at java.lang.StringBuffer.append(StringBuffer.java:270) ~[?:1.8.0_291]
从FE的jvm监控,发下下线的时候,堆内存比其他节点高,在90%以上。之后,就下线了。
之后,服务自动启动。