为了更快的定位您的问题,请提供以下信息,谢谢
【详述】问题详细描述
【背景】集群拉起来,还未做什么写入,就报错了。
【业务影响】
【是否存算分离】是
【StarRocks版本】例如:3.2.2
【集群规模】例如:3fe(1 follower+2observer)+3be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】邮箱:540945000@qq.com
【附件】
fe.log
2024-06-26 16:16:26,015 WARN (RepNode 192.168.98.14_7304_1719389743706(-1)|212) [BDBStateChangeListener.stateChange():79] this node is DETACHED
2024-06-26 16:16:36,012 INFO (UNKNOWN 192.168.98.14_7304_1719389743706(-1)|1) [BDBEnvironment.setupEnvironment():288] state change done, current role UNKNOWN
2024-06-26 16:16:36,013 ERROR (UNKNOWN 192.168.98.14_7304_1719389743706(-1)|1) [BDBEnvironment.setupEnvironment():305] failed to setup environment after retried 3 times
com.sleepycat.je.EnvironmentFailureException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.98.14_7304_1719389743706(-1):/data-sdd/install/starrocks/data/meta/bdb Clock delta: -7978 ms. between Feeder: 192.168.98.13_7304_1718700577526 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.98.14_7304_1719389743706(-1) Originally thrown by HA thread: RepNode 192.168.98.14_7304_1719389743706(-1)
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:291) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:175) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:153) ~[starrocks-fe.jar:?]
at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1181) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1130) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.start(StarRocksFE.java:135) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.98.14_7304_1719389743706(-1):/data-sdd/install/starrocks/data/meta/bdb Clock delta: -7978 ms. between Feeder: 192.168.98.13_7304_1718700577526 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.98.14_7304_1719389743706(-1) Originally thrown by HA thread: RepNode 192.168.98.14_7304_1719389743706(-1)
at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.checkClockSkew(ReplicaFeederHandshake.java:432) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.execute(ReplicaFeederHandshake.java:269) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.Replica.initReplicaLoop(Replica.java:709) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:485) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:412) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1869) ~[starrocks-bdb-je-18.3.16.jar:?]
2024-06-26 16:16:36,013 INFO (UNKNOWN 192.168.98.14_7304_1719389743706(-1)|1) [BDBEnvironment.close():517] start to close epoch database
2024-06-26 16:16:36,013 INFO (UNKNOWN 192.168.98.14_7304_1719389743706(-1)|1) [BDBEnvironment.close():526] close epoch database end
2024-06-26 16:16:36,013 INFO (UNKNOWN 192.168.98.14_7304_1719389743706(-1)|1) [BDBEnvironment.close():528] start to close replicated environment
2024-06-26 16:16:36,019 INFO (UNKNOWN 192.168.98.14_7304_1719389743706(-1)|1) [BDBEnvironment.close():538] close replicated environment end
2024-06-26 16:16:36,019 ERROR (UNKNOWN 192.168.98.14_7304_1719389743706(-1)|1) [StarRocksFE.start():192] StarRocksFE start failed
com.starrocks.journal.JournalException: failed to setup environment after retried 3 times
at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:306) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:175) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:153) ~[starrocks-fe.jar:?]
at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1181) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1130) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.start(StarRocksFE.java:135) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
Caused by: com.sleepycat.je.EnvironmentFailureException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.98.14_7304_1719389743706(-1):/data-sdd/install/starrocks/data/meta/bdb Clock delta: -7978 ms. between Feeder: 192.168.98.13_7304_1718700577526 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.98.14_7304_1719389743706(-1) Originally thrown by HA thread: RepNode 192.168.98.14_7304_1719389743706(-1)
at com.sleepycat.je.EnvironmentFailureException.wrapSelf(EnvironmentFailureException.java:230) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:291) ~[starrocks-fe.jar:?]
... 7 more
Caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.98.14_7304_1719389743706(-1):/data-sdd/install/starrocks/data/meta/bdb Clock delta: -7978 ms. between Feeder: 192.168.98.13_7304_1718700577526 and this Replica exceeds max permissible delta: 5000 ms. HANDSHAKE_ERROR: Error during the handshake between two nodes. Some validity or compatibility check failed, preventing further communication between the nodes. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.98.14_7304_1719389743706(-1) Originally thrown by HA thread: RepNode 192.168.98.14_7304_1719389743706(-1)
at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.checkClockSkew(ReplicaFeederHandshake.java:432) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.stream.ReplicaFeederHandshake.execute(ReplicaFeederHandshake.java:269) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.Replica.initReplicaLoop(Replica.java:709) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoopInternal(Replica.java:485) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.Replica.runReplicaLoop(Replica.java:412) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1869) ~[starrocks-bdb-je-18.3.16.jar:?]