fe挂掉无法重启

为了更快的定位您的问题,请提供以下信息,谢谢
【详述】fe挂掉了
【业务影响】是
【是否存算分离】否
【StarRocks版本】例如:3.2.11
【集群规模】例如:2fe+3be(fe与be混部)

【附件】
com.sleepycat.je.rep.InsufficientLogException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.rep.InsufficientLogException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.1.236_9010_1729493956533(3):/data/starrocks/fe_meta/bdb INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.1.236_9010_1729493956533(-1)refreshVLSN=-1 logProviders=[Node:192.168.1.208_9010_1728977720261 192.168.1.208:9010 (is member) changeVersion:1 LocalCBVLSN:-1 at:Tue Oct 15 15:35:20 GMT+08:00 2024 jeVersion:18.3.16
] repImpl=com.sleepycat.je.rep.impl.RepImpl@6f8f8a80 props={REFRESH_VLSN=-1, PORT=9010, HOSTNAME=172.16.214.1, NODE_NAME=192.168.1.236_9010_1729493956533, P_NODETYPE0=ELECTABLE, P_NODENAME0=192.168.1.208_9010_1728977720261, P_PORT0=9010, GROUP_NAME=PALO_JOURNAL_GROUP, P_HOSTNAME0=192.168.1.208, ENV_DIR=/data/starrocks/fe_meta/bdb, P_NUMPROVIDERS=1}
at com.sleepycat.je.rep.InsufficientLogException.wrapSelf(InsufficientLogException.java:340) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:288) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:172) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:150) ~[starrocks-fe.jar:?]
at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1235) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1172) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.start(StarRocksFE.java:134) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
Caused by: com.sleepycat.je.rep.InsufficientLogException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.1.236_9010_1729493956533(3):/data/starrocks/fe_meta/bdb INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.1.236_9010_1729493956533(-1)
at com.sleepycat.je.rep.impl.node.RepNode.findRestoreSuppliers(RepNode.java:1582) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.queryGroupForMembership(RepNode.java:1420) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1808) ~[starrocks-bdb-je-18.3.16.jar:?]
2025-02-17 18:26:56.434+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():513] start to close epoch database
2025-02-17 18:26:56.434+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():522] close epoch database end
2025-02-17 18:26:56.434+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():524] start to close replicated environment
2025-02-17 18:26:56.436+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():534] close replicated environment end
2025-02-17 18:27:01.437+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():266] start to setup bdb environment for 2 times
2025-02-17 18:27:01.839+08:00 WARN (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN
2025-02-17 18:27:01.840+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():277] replicated environment is all set, wait for state change…
2025-02-17 18:27:01.856+08:00 WARN (RepNode 192.168.1.236_9010_1729493956533(-1)|92) [BDBStateChangeListener.stateChange():79] this node is DETACHED
2025-02-17 18:27:11.841+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():285] state change done, current role UNKNOWN
2025-02-17 18:27:11.842+08:00 ERROR (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():302] failed to setup environment after retried 2 times
com.sleepycat.je.rep.InsufficientLogException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.rep.InsufficientLogException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.1.236_9010_1729493956533(3):/data/starrocks/fe_meta/bdb INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.1.236_9010_1729493956533(-1)refreshVLSN=-1 logProviders=[Node:192.168.1.208_9010_1728977720261 192.168.1.208:9010 (is member) changeVersion:1 LocalCBVLSN:-1 at:Tue Oct 15 15:35:20 GMT+08:00 2024 jeVersion:18.3.16
] repImpl=com.sleepycat.je.rep.impl.RepImpl@2d195ee4 props={REFRESH_VLSN=-1, PORT=9010, HOSTNAME=172.16.214.1, NODE_NAME=192.168.1.236_9010_1729493956533, P_NODETYPE0=ELECTABLE, P_NODENAME0=192.168.1.208_9010_1728977720261, P_PORT0=9010, GROUP_NAME=PALO_JOURNAL_GROUP, P_HOSTNAME0=192.168.1.208, ENV_DIR=/data/starrocks/fe_meta/bdb, P_NUMPROVIDERS=1}
at com.sleepycat.je.rep.InsufficientLogException.wrapSelf(InsufficientLogException.java:340) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:288) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:172) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:150) ~[starrocks-fe.jar:?]
at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1235) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1172) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.start(StarRocksFE.java:134) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
Caused by: com.sleepycat.je.rep.InsufficientLogException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.1.236_9010_1729493956533(3):/data/starrocks/fe_meta/bdb INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.1.236_9010_1729493956533(-1)
at com.sleepycat.je.rep.impl.node.RepNode.findRestoreSuppliers(RepNode.java:1582) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.queryGroupForMembership(RepNode.java:1420) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1808) ~[starrocks-bdb-je-18.3.16.jar:?]
2025-02-17 18:27:13.711+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():513] start to close epoch database
2025-02-17 18:27:13.712+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():522] close epoch database end
2025-02-17 18:27:13.712+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():524] start to close replicated environment
2025-02-17 18:27:13.713+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():534] close replicated environment end
2025-02-17 18:27:18.713+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():266] start to setup bdb environment for 3 times
2025-02-17 18:27:18.939+08:00 WARN (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN
2025-02-17 18:27:18.939+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():277] replicated environment is all set, wait for state change…
2025-02-17 18:27:18.952+08:00 WARN (RepNode 192.168.1.236_9010_1729493956533(-1)|117) [BDBStateChangeListener.stateChange():79] this node is DETACHED
2025-02-17 18:27:28.941+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():285] state change done, current role UNKNOWN
2025-02-17 18:27:28.942+08:00 ERROR (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():302] failed to setup environment after retried 3 times
com.sleepycat.je.rep.InsufficientLogException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.rep.InsufficientLogException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.1.236_9010_1729493956533(3):/data/starrocks/fe_meta/bdb INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.1.236_9010_1729493956533(-1)refreshVLSN=-1 logProviders=[Node:192.168.1.208_9010_1728977720261 192.168.1.208:9010 (is member) changeVersion:1 LocalCBVLSN:-1 at:Tue Oct 15 15:35:20 GMT+08:00 2024 jeVersion:18.3.16
] repImpl=com.sleepycat.je.rep.impl.RepImpl@5560bcdf props={REFRESH_VLSN=-1, PORT=9010, HOSTNAME=172.16.214.1, NODE_NAME=192.168.1.236_9010_1729493956533, P_NODETYPE0=ELECTABLE, P_NODENAME0=192.168.1.208_9010_1728977720261, P_PORT0=9010, GROUP_NAME=PALO_JOURNAL_GROUP, P_HOSTNAME0=192.168.1.208, ENV_DIR=/data/starrocks/fe_meta/bdb, P_NUMPROVIDERS=1}
at com.sleepycat.je.rep.InsufficientLogException.wrapSelf(InsufficientLogException.java:340) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:288) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:172) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:150) ~[starrocks-fe.jar:?]
at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1235) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1172) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.start(StarRocksFE.java:134) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
Caused by: com.sleepycat.je.rep.InsufficientLogException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.1.236_9010_1729493956533(3):/data/starrocks/fe_meta/bdb INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.1.236_9010_1729493956533(-1)
at com.sleepycat.je.rep.impl.node.RepNode.findRestoreSuppliers(RepNode.java:1582) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.queryGroupForMembership(RepNode.java:1420) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1808) ~[starrocks-bdb-je-18.3.16.jar:?]
2025-02-17 18:27:29.193+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():513] start to close epoch database
2025-02-17 18:27:29.193+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():522] close epoch database end
2025-02-17 18:27:29.193+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():524] start to close replicated environment
2025-02-17 18:27:29.194+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.close():534] close replicated environment end
2025-02-17 18:27:29.194+08:00 ERROR (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [StarRocksFE.start():191] StarRocksFE start failed
com.starrocks.journal.JournalException: failed to setup environment after retried 3 times
at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:303) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setup(BDBEnvironment.java:172) ~[starrocks-fe.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.initBDBEnvironment(BDBEnvironment.java:150) ~[starrocks-fe.jar:?]
at com.starrocks.journal.JournalFactory.create(JournalFactory.java:31) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initJournal(GlobalStateMgr.java:1235) ~[starrocks-fe.jar:?]
at com.starrocks.server.GlobalStateMgr.initialize(GlobalStateMgr.java:1172) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.start(StarRocksFE.java:134) ~[starrocks-fe.jar:?]
at com.starrocks.StarRocksFE.main(StarRocksFE.java:84) ~[starrocks-fe.jar:?]
Caused by: com.sleepycat.je.rep.InsufficientLogException: (JE 18.3.16) Environment must be closed, caused by: com.sleepycat.je.rep.InsufficientLogException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.1.236_9010_1729493956533(3):/data/starrocks/fe_meta/bdb INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.1.236_9010_1729493956533(-1)refreshVLSN=-1 logProviders=[Node:192.168.1.208_9010_1728977720261 192.168.1.208:9010 (is member) changeVersion:1 LocalCBVLSN:-1 at:Tue Oct 15 15:35:20 GMT+08:00 2024 jeVersion:18.3.16
] repImpl=com.sleepycat.je.rep.impl.RepImpl@5560bcdf props={REFRESH_VLSN=-1, PORT=9010, HOSTNAME=172.16.214.1, NODE_NAME=192.168.1.236_9010_1729493956533, P_NODETYPE0=ELECTABLE, P_NODENAME0=192.168.1.208_9010_1728977720261, P_PORT0=9010, GROUP_NAME=PALO_JOURNAL_GROUP, P_HOSTNAME0=192.168.1.208, ENV_DIR=/data/starrocks/fe_meta/bdb, P_NUMPROVIDERS=1}
at com.sleepycat.je.rep.InsufficientLogException.wrapSelf(InsufficientLogException.java:340) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkIfInvalid(EnvironmentImpl.java:1835) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.dbi.EnvironmentImpl.checkOpen(EnvironmentImpl.java:1844) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.checkOpen(Environment.java:2697) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.Environment.openDatabase(Environment.java:659) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.starrocks.journal.bdbje.BDBEnvironment.setupEnvironment(BDBEnvironment.java:288) ~[starrocks-fe.jar:?]
… 7 more
Caused by: com.sleepycat.je.rep.InsufficientLogException: Environment invalid because of previous exception: (JE 18.3.16) 192.168.1.236_9010_1729493956533(3):/data/starrocks/fe_meta/bdb INSUFFICIENT_LOG: Log files at this node are obsolete. Environment is invalid and must be closed. Originally thrown by HA thread: RepNode 192.168.1.236_9010_1729493956533(-1)
at com.sleepycat.je.rep.impl.node.RepNode.findRestoreSuppliers(RepNode.java:1582) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.queryGroupForMembership(RepNode.java:1420) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1808) ~[starrocks-bdb-je-18.3.16.jar:?]
2025-02-17 18:44:38.557+08:00 INFO (main|1) [StarRocksFE.start():128] StarRocks FE starting, version: 3.2.11-10a5f0e
2025-02-17 18:44:38.564+08:00 INFO (main|1) [FrontendOptions.initAddrUseIp():249] Use IP init local addr, IP: /172.16.214.1
2025-02-17 18:44:38.642+08:00 INFO (main|1) [ConsistencyChecker.initWorkTime():121] parsed startDate: 1970-01-01T23:00:00.000+0800, endDate: 1970-01-01T04:00:00.000+0800
2025-02-17 18:44:38.874+08:00 INFO (main|1) [Auth.grantRoleInternal():837] grant operator to ‘root’@’%’, isReplay = true
2025-02-17 18:44:38.901+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():283] create built-in role root[-1]
2025-02-17 18:44:38.906+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():283] create built-in role db_admin[-2]
2025-02-17 18:44:38.906+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():283] create built-in role cluster_admin[-3]
2025-02-17 18:44:38.907+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():283] create built-in role user_admin[-4]
2025-02-17 18:44:38.907+08:00 INFO (main|1) [AuthorizationMgr.initBuiltinRoleUnlocked():283] create built-in role public[-5]
2025-02-17 18:44:38.907+08:00 INFO (main|1) [GlobalStateMgr.initAuth():1211] using new privilege framework…
2025-02-17 18:44:39.109+08:00 INFO (main|1) [NodeMgr.getHelperNodes():697] get helper nodes: [172.16.214.1:9010]
2025-02-17 18:44:39.123+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():498] Current run_mode is shared_nothing
2025-02-17 18:44:39.123+08:00 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():505] Got cluster id: 1369338607, role: LEADER, node name: 192.168.1.236_9010_1729493956533 and run_mode: shared_nothing
2025-02-17 18:44:39.124+08:00 INFO (main|1) [BDBEnvironment.ensureHelperInLocal():331] skip check local environment for observer
2025-02-17 18:44:39.140+08:00 INFO (main|1) [BDBEnvironment.setupEnvironment():266] start to setup bdb environment for 1 times
2025-02-17 18:44:39.824+08:00 WARN (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN
2025-02-17 18:44:39.870+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():277] replicated environment is all set, wait for state change…
2025-02-17 18:44:49.871+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():285] state change done, current role UNKNOWN
2025-02-17 18:44:49.877+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BDBEnvironment.setupEnvironment():289] end setup bdb environment after 1 times
2025-02-17 18:44:49.884+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1583] start load image from /data/starrocks/fe_meta/image/image.22448109. is ckpt: false
2025-02-17 18:44:49.922+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadHeaderV2():1832] finished to replay header from image
2025-02-17 18:44:49.936+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BackendCoreStat.setNumOfHardwareCoresOfBe():38] set numOfHardwareCores of be(10723) to 16, current cpuCores stats: {10723=16}
2025-02-17 18:44:49.936+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BackendCoreStat.setNumOfHardwareCoresOfBe():38] set numOfHardwareCores of be(59000) to 8, current cpuCores stats: {10723=16, 59000=8}
2025-02-17 18:44:49.937+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [BackendCoreStat.setNumOfHardwareCoresOfBe():38] set numOfHardwareCores of be(61868) to 8, current cpuCores stats: {10723=16, 59000=8, 61868=8}
2025-02-17 18:44:49.938+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 1 from image
2025-02-17 18:44:54.691+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [MaterializedView.setActive():495] set test_wx_daily_adc_reporting to active
2025-02-17 18:44:54.811+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [CachingMvPlanContextBuilder.putAstIfAbsent():183] Add mv test_wx_daily_adc_reporting input ast cache
2025-02-17 18:44:54.812+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [MaterializedView.setActive():495] set dws_wx_app_roi_trace_ltv to active
2025-02-17 18:44:54.892+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [CachingMvPlanContextBuilder.putAstIfAbsent():183] Add mv dws_wx_app_roi_trace_ltv input ast cache
2025-02-17 18:44:54.893+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [MaterializedView.setActive():495] set dws_wx_daily_conversion to active
2025-02-17 18:44:54.912+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [CachingMvPlanContextBuilder.putAstIfAbsent():183] Add mv dws_wx_daily_conversion input ast cache
2025-02-17 18:44:54.913+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [MaterializedView.setActive():495] set dws_wx_daily_benefit to active
2025-02-17 18:44:54.916+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [CachingMvPlanContextBuilder.putAstIfAbsent():183] Add mv dws_wx_daily_benefit input ast cache
2025-02-17 18:44:55.042+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [MaterializedView.setActive():495] set sr_report_view_asynchronization to active
2025-02-17 18:44:55.051+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [CachingMvPlanContextBuilder.putAstIfAbsent():183] Add mv sr_report_view_asynchronization input ast cache
2025-02-17 18:44:55.352+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 2 from image
2025-02-17 18:44:55.352+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 3 from image
2025-02-17 18:44:55.357+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 4 from image
2025-02-17 18:44:55.368+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 5 from image
2025-02-17 18:44:55.370+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 6 from image
2025-02-17 18:44:55.371+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 7 from image
2025-02-17 18:44:55.377+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 8 from image
2025-02-17 18:44:55.380+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 9 from image
2025-02-17 18:44:55.576+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 10 from image
2025-02-17 18:44:55.584+08:00 WARN (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [ColocateTableIndex.cleanupInvalidDbOrTable():1037] remove 0 invalid tableid: []
2025-02-17 18:44:55.585+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [ColocateTableIndex.loadColocateTableIndexV2():833] finished replay colocateTableIndex from image
2025-02-17 18:44:55.585+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 11 from image
2025-02-17 18:44:55.598+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 76032. current callback size: 1
2025-02-17 18:44:55.600+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 630923. current callback size: 2
2025-02-17 18:44:55.603+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1619228. current callback size: 3
2025-02-17 18:44:55.620+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1630620. current callback size: 4
2025-02-17 18:44:55.620+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 75910. current callback size: 5
2025-02-17 18:44:55.621+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1888020. current callback size: 6
2025-02-17 18:44:55.623+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1034244. current callback size: 7
2025-02-17 18:44:55.624+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1005980. current callback size: 8
2025-02-17 18:44:55.625+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 438546. current callback size: 9
2025-02-17 18:44:55.626+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 632604. current callback size: 10
2025-02-17 18:44:55.627+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 82838. current callback size: 11
2025-02-17 18:44:55.629+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1051655. current callback size: 12
2025-02-17 18:44:55.635+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 2422073. current callback size: 13
2025-02-17 18:44:55.636+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 492825. current callback size: 14
2025-02-17 18:44:55.637+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 80930. current callback size: 15
2025-02-17 18:44:55.638+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1198129. current callback size: 16
2025-02-17 18:44:55.640+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1490225. current callback size: 17
2025-02-17 18:44:55.640+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 80937. current callback size: 18
2025-02-17 18:44:55.641+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 439468. current callback size: 19
2025-02-17 18:44:55.641+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 80555. current callback size: 20
2025-02-17 18:44:55.642+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 660385. current callback size: 21
2025-02-17 18:44:55.643+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1216569. current callback size: 22
2025-02-17 18:44:55.643+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1737771. current callback size: 23
2025-02-17 18:44:55.644+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1676458. current callback size: 24
2025-02-17 18:44:55.645+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 2360208. current callback size: 25
2025-02-17 18:44:55.646+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 2438289. current callback size: 26
2025-02-17 18:44:55.648+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 600892. current callback size: 27
2025-02-17 18:44:55.649+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 2406545. current callback size: 28
2025-02-17 18:44:55.650+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1630625. current callback size: 29
2025-02-17 18:44:55.650+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 659888. current callback size: 30
2025-02-17 18:44:55.651+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1520429. current callback size: 31
2025-02-17 18:44:55.652+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 75706. current callback size: 32
2025-02-17 18:44:55.653+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 2283617. current callback size: 33
2025-02-17 18:44:55.654+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1363408. current callback size: 34
2025-02-17 18:44:55.654+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 2652911. current callback size: 35
2025-02-17 18:44:55.655+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1649365. current callback size: 36
2025-02-17 18:44:55.656+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1598933. current callback size: 37
2025-02-17 18:44:55.657+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 80847. current callback size: 38
2025-02-17 18:44:55.658+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1076800. current callback size: 39
2025-02-17 18:44:55.658+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 2261746. current callback size: 40
2025-02-17 18:44:55.659+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1061826. current callback size: 41
2025-02-17 18:44:55.660+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1440455. current callback size: 42
2025-02-17 18:44:55.661+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1598924. current callback size: 43
2025-02-17 18:44:55.662+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1717324. current callback size: 44
2025-02-17 18:44:55.663+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1381581. current callback size: 45
2025-02-17 18:44:55.663+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 939604. current callback size: 46
2025-02-17 18:44:55.664+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 2674163. current callback size: 47
2025-02-17 18:44:55.664+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 76636. current callback size: 48
2025-02-17 18:44:55.665+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 894034. current callback size: 49
2025-02-17 18:44:55.665+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 80608. current callback size: 50
2025-02-17 18:44:55.666+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 712427. current callback size: 51
2025-02-17 18:44:55.666+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1400055. current callback size: 52
2025-02-17 18:44:55.667+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 549868. current callback size: 53
2025-02-17 18:44:55.668+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 2384329. current callback size: 54
2025-02-17 18:44:55.669+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 1109502. current callback size: 55
2025-02-17 18:44:55.670+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 2428885. current callback size: 56
2025-02-17 18:44:55.671+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 736368. current callback size: 57
2025-02-17 18:44:55.671+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [TxnStateCallbackFactory.addCallback():37] add callback of txn state : 679286. current callback size: 58
2025-02-17 18:44:55.671+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 12 from image
2025-02-17 18:44:55.984+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 13 from image
2025-02-17 18:44:55.984+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 14 from image
2025-02-17 18:44:55.994+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 15 from image
2025-02-17 18:44:55.995+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 16 from image
2025-02-17 18:44:56.160+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 17 from image
2025-02-17 18:44:56.172+08:00 INFO (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [GlobalStateMgr.loadImage():1651] Success load StarRocks meta block 18 from image
2025-02-17 18:44:56.173+08:00 WARN (UNKNOWN 192.168.1.236_9010_1729493956533(-1)|1) [AuthenticationProviderFactory.installPlugin():32] Plugin MYSQL_NATI

可以参考:https://docs.starrocks.io/zh/docs/administration/Meta_recovery/#7-最终应急方案 这里的文档做恢复