fe关闭后无法启动

💬 StarRocks 用户问答
#1

【详述】fe集群无法启动，端口无监听
【背景】操作过物化视图，删除正常表失败，后关闭无法启动
【业务影响】
【StarRocks版本】3.0.4
【集群规模】3fe（2 follower+1observer）+7be（fe与be混部）
【机器信息】CPU虚拟核/内存/网卡，例如：48C/128G/万兆

2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_ads_out_mjzyspb_d(11337126) to invalid because the table null was not exist.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_dws_amt_mjz_fysr_m(16219626) to invalid because the table null was not exist.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_ads_in_ss_swrc_m(16373198) to invalid because the table null was not exist.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_ads_in_ss_swrc_d(16373139) to invalid because the table null was not exist.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_dws_in_ksfz_amount_d(11335121) to invalid because the table null was not exist.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_ads_in_leave_hldj_m(13817172) to invalid because the table null was not exist.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_ads_in_leave_hldj_d(13817143) to invalid because the table null was not exist.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_dws_in_ksfz_amount_m(11334998) to invalid because the table null was not exist.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1583] Setting the materialized view t_ads_ylzl_dbz_zbzdzrc(16055561) to invalid because the materialized viewt_is_dict_dbz(16049914) is invalid.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_is_dict_dbz(16049914) to invalid because the table null was not exist.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_dws_in_drug_amount_d(11827073) to invalid because the table null was not exist.
2023-08-23 21:26:01,815 WARN (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.updateBaseTableRelatedMv():1576] Setting the materialized view t_dws_in_ysjbyb_czqk_m(15515536) to invalid because the table null was not exist.
2023-08-23 21:26:16,369 WARN (RepNode 192.168.2.71_9010_1670321150475(-1)|69) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: LEADER
2023-08-23 21:26:18,658 WARN (stateChangeExecutor|83) [GlobalStateMgr.replayJournalInner():2156] catch exception when replaying 137452900,
com.google.gson.JsonSyntaxException: duplicate key: 7417179
        at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:190) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145) ~[spark-dpp-1.0.0.jar:?]
        at com.starrocks.persist.gson.GsonUtils$ProcessHookTypeAdapterFactory$1.read(GsonUtils.java:641) ~[starrocks-fe.jar:?]
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[spark-dpp-1.0.0.jar:?]
        at com.starrocks.persist.gson.GsonUtils$ProcessHookTypeAdapterFactory$1.read(GsonUtils.java:641) ~[starrocks-fe.jar:?]
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[spark-dpp-1.0.0.jar:?]
        at com.starrocks.persist.gson.GsonUtils$ProcessHookTypeAdapterFactory$1.read(GsonUtils.java:641) ~[starrocks-fe.jar:?]
        at com.google.gson.Gson.fromJson(Gson.java:963) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.Gson.fromJson(Gson.java:928) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.Gson.fromJson(Gson.java:877) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.Gson.fromJson(Gson.java:848) ~[spark-dpp-1.0.0.jar:?]
at com.google.gson.Gson.fromJson(Gson.java:848) ~[spark-dpp-1.0.0.jar:?]
        at com.starrocks.persist.ChangeMaterializedViewRefreshSchemeLog.read(ChangeMaterializedViewRefreshSchemeLog.java:71) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.JournalEntity.readFields(JournalEntity.java:358) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBJournalCursor.deserializeData(BDBJournalCursor.java:251) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBJournalCursor.next(BDBJournalCursor.java:295) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.replayJournalInner(GlobalStateMgr.java:2137) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.replayJournal(GlobalStateMgr.java:2097) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.transferToLeader(GlobalStateMgr.java:1142) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.access$100(GlobalStateMgr.java:324) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr$1.transferToLeader(GlobalStateMgr.java:721) ~[starrocks-fe.jar:?]
        at com.starrocks.ha.StateChangeExecutor.runOneCycle(StateChangeExecutor.java:103) ~[starrocks-fe.jar:?]
        at com.starrocks.common.util.Daemon.run(Daemon.java:115) ~[starrocks-fe.jar:?]
2023-08-23 21:26:18,663 ERROR (stateChangeExecutor|83) [GlobalStateMgr.transferToLeader():1147] failed to init journal after transfer to leader! will exit
com.google.gson.JsonSyntaxException: duplicate key: 7417179
        at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:190) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145) ~[spark-dpp-1.0.0.jar:?]
        at com.starrocks.persist.gson.GsonUtils$ProcessHookTypeAdapterFactory$1.read(GsonUtils.java:641) ~[starrocks-fe.jar:?]
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[spark-dpp-1.0.0.jar:?]
        at com.starrocks.persist.gson.GsonUtils$ProcessHookTypeAdapterFactory$1.read(GsonUtils.java:641) ~[starrocks-fe.jar:?]
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[spark-dpp-1.0.0.jar:?]
        at com.starrocks.persist.gson.GsonUtils$ProcessHookTypeAdapterFactory$1.read(GsonUtils.java:641) ~[starrocks-fe.jar:?]
        at com.google.gson.Gson.fromJson(Gson.java:963) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.Gson.fromJson(Gson.java:928) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.Gson.fromJson(Gson.java:877) ~[spark-dpp-1.0.0.jar:?]
        at com.google.gson.Gson.fromJson(Gson.java:848) ~[spark-dpp-1.0.0.jar:?]
        at com.starrocks.persist.ChangeMaterializedViewRefreshSchemeLog.read(ChangeMaterializedViewRefreshSchemeLog.java:71) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.JournalEntity.readFields(JournalEntity.java:358) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBJournalCursor.deserializeData(BDBJournalCursor.java:251) ~[starrocks-fe.jar:?]
        at com.starrocks.journal.bdbje.BDBJournalCursor.next(BDBJournalCursor.java:295) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.replayJournalInner(GlobalStateMgr.java:2137) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.replayJournal(GlobalStateMgr.java:2097) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.transferToLeader(GlobalStateMgr.java:1142) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr.access$100(GlobalStateMgr.java:324) ~[starrocks-fe.jar:?]
        at com.starrocks.server.GlobalStateMgr$1.transferToLeader(GlobalStateMgr.java:721) ~[starrocks-fe.jar:?]
        at com.starrocks.ha.StateChangeExecutor.runOneCycle(StateChangeExecutor.java:103) ~[starrocks-fe.jar:?]
        at com.starrocks.common.util.Daemon.run(Daemon.java:115) ~[starrocks-fe.jar:?]
#2

vx 1657602736
加过小鳄

#3

2023-08-23 22:35:09,294 INFO (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.waitForReady():1102] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2023-08-23 22:35:11,294 INFO (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.waitForReady():1102] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2023-08-23 22:35:13,295 INFO (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.waitForReady():1102] wait globalStateMgr to be ready. FE type: INIT. is ready: false
2023-08-23 22:35:15,295 INFO (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.waitForReady():1102] wait globalStateMgr to be ready. FE type: INIT. is ready: false

#4

ip是对的…

#5

fe内存给到了32g

#6

可以fe配置里加入ignore_materialized_view_error=true 启动，修复版本我这边确认下，另外您的节点，建议3个follower实现高可用

#7

下个3.0的最新版本会修复该问题，发布后您可以升级

#8

ok十分感谢大佬回复

#9

加了还是启动不了，还是这个报错

#10 
2023-08-24 10:11:38,653 ERROR (replayer|91) [GlobalStateMgr$5.runOneCycle():2012] replayer thread catch an exception when replay journal 137452900.
com.google.gson.JsonSyntaxException: duplicate key: 7417179
	at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:190) ~[spark-dpp-1.0.0.jar:?]
	at com.google.gson.internal.bind.MapTypeAdapterFactory$Adapter.read(MapTypeAdapterFactory.java:145) ~[spark-dpp-1.0.0.jar:?]
	at com.starrocks.persist.gson.GsonUtils$ProcessHookTypeAdapterFactory$1.read(GsonUtils.java:641) ~[starrocks-fe.jar:?]
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[spark-dpp-1.0.0.jar:?]
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[spark-dpp-1.0.0.jar:?]
	at com.starrocks.persist.gson.GsonUtils$ProcessHookTypeAdapterFactory$1.read(GsonUtils.java:641) ~[starrocks-fe.jar:?]
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$1.read(ReflectiveTypeAdapterFactory.java:131) ~[spark-dpp-1.0.0.jar:?]
	at com.google.gson.internal.bind.ReflectiveTypeAdapterFactory$Adapter.read(ReflectiveTypeAdapterFactory.java:222) ~[spark-dpp-1.0.0.jar:?]
	at com.starrocks.persist.gson.GsonUtils$ProcessHookTypeAdapterFactory$1.read(GsonUtils.java:641) ~[starrocks-fe.jar:?]
	at com.google.gson.Gson.fromJson(Gson.java:963) ~[spark-dpp-1.0.0.jar:?]
	at com.google.gson.Gson.fromJson(Gson.java:928) ~[spark-dpp-1.0.0.jar:?]
	at com.google.gson.Gson.fromJson(Gson.java:877) ~[spark-dpp-1.0.0.jar:?]
	at com.google.gson.Gson.fromJson(Gson.java:848) ~[spark-dpp-1.0.0.jar:?]
	at com.starrocks.persist.ChangeMaterializedViewRefreshSchemeLog.read(ChangeMaterializedViewRefreshSchemeLog.java:71) ~[starrocks-fe.jar:?]
	at com.starrocks.journal.JournalEntity.readFields(JournalEntity.java:358) ~[starrocks-fe.jar:?]
	at com.starrocks.journal.bdbje.BDBJournalCursor.deserializeData(BDBJournalCursor.java:251) ~[starrocks-fe.jar:?]
	at com.starrocks.journal.bdbje.BDBJournalCursor.next(BDBJournalCursor.java:295) ~[starrocks-fe.jar:?]
	at com.starrocks.server.GlobalStateMgr.replayJournalInner(GlobalStateMgr.java:2137) ~[starrocks-fe.jar:?]
	at com.starrocks.server.GlobalStateMgr$5.runOneCycle(GlobalStateMgr.java:2003) ~[starrocks-fe.jar:?]
	at com.starrocks.common.util.Daemon.run(Daemon.java:115) ~[starrocks-fe.jar:?]
	at com.starrocks.server.GlobalStateMgr$5.run(GlobalStateMgr.java:2068) ~[starrocks-fe.jar:?]
2023-08-24 10:11:39,617 INFO (UNKNOWN 192.168.2.71_9010_1670321150475(-1)|1) [GlobalStateMgr.waitForReady():1102] wait globalStateMgr to be ready. FE type: UNKNOWN. is ready: false
#11

不好意思，这个是2.5.10+fe配置里加入ignore_materialized_view_error=true可以绕过，给您这边单独打了个修复patch的fe包，您替换启动下http://starrocks-public.oss-cn-zhangjiakou.aliyuncs.com/starrocks-fe-3.0.4-with_mv_dup_patch.jar。替换下fe安装目录，lib下的starrocks-fe.jar

#12

感谢，启动成功了

#13

感谢大佬，环境能启动了

1赞
#14

你好，我也出现了，这个问题，没看到这个帖子。我是直接升级了2.5.17，但是升级之后出现动态分区表异常。然后我就把表rename了，但是元数据还存在。现在一直删不掉。

企业微信截图_17047710353442
企业微信截图_170477103534422560×1400 725 KB

#15

@dongquan 大佬求助

京ICP备2022026421号-1