【详述】fe节点异常退出
【背景】无
【StarRocks版本】例如:1.19.3
【集群规模】例如:2fe(1 follower+1observer)+3be(fe独立部署)
【机器信息】CPU虚拟核/内存/网卡,例如:8C/32G
fe.log:
2022-01-11 09:44:20,876 WARN (heartbeat-mgr-pool-5|196) [Util.getResultForUrl():337] failed to get result from url: http://172.31.0.136:8030/api/bootstrap?cluster_id=705545127&token=fa0bb3a9-f706-4873-ab04-31393
4a18530. connect timed out
2022-01-11 09:44:20,877 WARN (heartbeat mgr|22) [HeartbeatMgr.runAfterCatalogReady():142] get bad heartbeat response: type: FRONTEND, status: BAD, msg: got exception, name: 172.31.0.136_9010_1639356434637, query
Port: 0, rpcPort: 0, replayedJournalId: 0
2022-01-11 09:44:20,937 INFO (starrocks-mysql-nio-pool-74800|233757) [StmtExecutor.execute():451] execute Exception. Unknown table 'f_jn_ad_campaign_d
2022-01-11 09:44:21,425 WARN (Routine load scheduler|34) [KafkaUtil$ProxyAPI.sendProxyRequest():169] failed to send proxy request.
java.util.concurrent.ExecutionException: Ocurrs time out with specfied time 5 SECONDS
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy$2.get(ProtobufRpcProxy.java:578) ~[jprotobuf-rpc-core-4.1.8.jar:?]
at com.starrocks.common.util.KafkaUtil$ProxyAPI.sendProxyRequest(KafkaUtil.java:161) [starrocks-fe.jar:?]
at com.starrocks.common.util.KafkaUtil$ProxyAPI.getAllKafkaPartitions(KafkaUtil.java:94) [starrocks-fe.jar:?]
at com.starrocks.common.util.KafkaUtil.getAllKafkaPartitions(KafkaUtil.java:56) [starrocks-fe.jar:?]
at com.starrocks.load.routineload.KafkaRoutineLoadJob.getAllKafkaPartitions(KafkaRoutineLoadJob.java:362) [starrocks-fe.jar:?]
at com.starrocks.load.routineload.KafkaRoutineLoadJob.unprotectNeedReschedule(KafkaRoutineLoadJob.java:291) [starrocks-fe.jar:?]
at com.starrocks.load.routineload.RoutineLoadJob.update(RoutineLoadJob.java:1185) [starrocks-fe.jar:?]
at com.starrocks.load.routineload.RoutineLoadManager.updateRoutineLoadJob(RoutineLoadManager.java:531) [starrocks-fe.jar:?]
at com.starrocks.load.routineload.RoutineLoadScheduler.process(RoutineLoadScheduler.java:67) [starrocks-fe.jar:?]
at com.starrocks.load.routineload.RoutineLoadScheduler.runAfterCatalogReady(RoutineLoadScheduler.java:59) [starrocks-fe.jar:?]
at com.starrocks.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:61) [starrocks-fe.jar:?]
at com.starrocks.common.util.Daemon.run(Daemon.java:119) [starrocks-fe.jar:?]
Caused by: java.util.concurrent.TimeoutException: Ocurrs time out with specfied time 5 SECONDS
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.doWaitCallback(ProtobufRpcProxy.java:625) ~[jprotobuf-rpc-core-4.1.8.jar:?]
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.access$000(ProtobufRpcProxy.java:54) ~[jprotobuf-rpc-core-4.1.8.jar:?]
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy$2.get(ProtobufRpcProxy.java:576) ~[jprotobuf-rpc-core-4.1.8.jar:?]
… 11 more
2022-01-11 09:44:21,425 WARN (Routine load scheduler|34) [KafkaRoutineLoadJob.unprotectNeedReschedule():294] ROUTINE_LOAD_JOB=10498627, error_msg={Job failed to fetch all current partition with error [Failed to
send proxy request: Ocurrs time out with specfied time 5 SECONDS]}
com.starrocks.common.LoadException: Failed to send proxy request: Ocurrs time out with specfied time 5 SECONDS
at com.starrocks.common.util.KafkaUtil$ProxyAPI.sendProxyRequest(KafkaUtil.java:170) ~[starrocks-fe.jar:?]
at com.starrocks.common.util.KafkaUtil$ProxyAPI.getAllKafkaPartitions(KafkaUtil.java:94) ~[starrocks-fe.jar:?]
at com.starrocks.common.util.KafkaUtil.getAllKafkaPartitions(KafkaUtil.java:56) ~[starrocks-fe.jar:?]
at com.starrocks.load.routineload.KafkaRoutineLoadJob.getAllKafkaPartitions(KafkaRoutineLoadJob.java:362) ~[starrocks-fe.jar:?]
at com.starrocks.load.routineload.KafkaRoutineLoadJob.unprotectNeedReschedule(KafkaRoutineLoadJob.java:291) [starrocks-fe.jar:?]
at com.starrocks.load.routineload.RoutineLoadJob.update(RoutineLoadJob.java:1185) [starrocks-fe.jar:?]
at com.starrocks.load.routineload.RoutineLoadManager.updateRoutineLoadJob(RoutineLoadManager.java:531) [starrocks-fe.jar:?]
at com.starrocks.load.routineload.RoutineLoadScheduler.process(RoutineLoadScheduler.java:67) [starrocks-fe.jar:?]
at com.starrocks.load.routineload.RoutineLoadScheduler.runAfterCatalogReady(RoutineLoadScheduler.java:59) [starrocks-fe.jar:?]
at com.starrocks.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:61) [starrocks-fe.jar:?]
at com.starrocks.common.util.Daemon.run(Daemon.java:119) [starrocks-fe.jar:?]
2022-01-11 09:44:21,431 INFO (Routine load scheduler|34) [RoutineLoadScheduler.process():76] there are 0 job need schedule
2022-01-11 09:44:21,872 INFO (starrocks-mysql-nio-pool-74800|233757) [StmtExecutor.execute():451] execute Exception. Unknown table 'f_jn_ad_campaign_d
你好,看报错应该是routine load任务超时了。fe挂起的原因可以看下fe.out有内容吗