为了更快的定位您的问题,请提供以下信息,谢谢
【详述】大量 failed to find any backend, needAvailable=true日志记录,导致问题:大量查询无法正常进入到CN节点处理,90%查询比正常状态缓慢,试过重启CN节点后正常一段时间后又重现该情况
【背景】正常运行,可能有部分大查询
【业务影响】
【是否存算分离】是
【StarRocks版本】3.3.5
【集群规模】例如:3fe + 7cn(fe与cn均独立部署)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/148G/万兆
【联系方式】tanheyuan@outlook.com
【附件】
- fe.warn.log
2025-05-14 10:32:25.530+08:00 WARN (nioEventLoopGroup-6-62|290) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:32:25.773+08:00 WARN (nioEventLoopGroup-6-6|171) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:32:26.381+08:00 WARN (nioEventLoopGroup-6-14|180) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:32:26.447+08:00 WARN (nioEventLoopGroup-6-20|186) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:32:26.686+08:00 WARN (nioEventLoopGroup-6-30|210) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:32:27.018+08:00 WARN (nioEventLoopGroup-6-38|218) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:32:30.725+08:00 WARN (nioEventLoopGroup-6-49|253) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:32:30.726+08:00 WARN (nioEventLoopGroup-6-50|278) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:32:30.798+08:00 WARN (nioEventLoopGroup-6-58|286) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:32:31.040+08:00 WARN (nioEventLoopGroup-6-8|174) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:26.272+08:00 WARN (nioEventLoopGroup-6-30|210) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:26.562+08:00 WARN (nioEventLoopGroup-6-38|218) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:27.338+08:00 WARN (nioEventLoopGroup-6-46|249) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:27.381+08:00 WARN (nioEventLoopGroup-6-50|278) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:27.830+08:00 WARN (nioEventLoopGroup-6-58|286) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:31.694+08:00 WARN (nioEventLoopGroup-6-8|174) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:32.303+08:00 WARN (nioEventLoopGroup-6-17|183) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:33.125+08:00 WARN (nioEventLoopGroup-6-25|195) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:33.226+08:00 WARN (nioEventLoopGroup-6-29|209) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:33.345+08:00 WARN (nioEventLoopGroup-6-37|217) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:37.198+08:00 WARN (nioEventLoopGroup-6-49|253) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:37.963+08:00 WARN (nioEventLoopGroup-6-57|285) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:38.523+08:00 WARN (nioEventLoopGroup-6-1|165) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:38.621+08:00 WARN (nioEventLoopGroup-6-5|170) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:42.063+08:00 WARN (nioEventLoopGroup-6-18|184) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:47.511+08:00 WARN (nioEventLoopGroup-6-27|200) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:33:52.464+08:00 WARN (starrocks-mysql-nio-pool-23941|1439896) [PruneComplexTypeUtil$Context.checkCanPrune():172] Complex type columnRefOperator[ARRAY<VARCHAR(65533)>] and scalarOperator[ARRAY] should has the same type
2025-05-14 10:33:52.464+08:00 WARN (starrocks-mysql-nio-pool-23941|1439896) [PruneComplexTypeUtil$Context.checkCanPrune():172] Complex type columnRefOperator[ARRAY<VARCHAR(65533)>] and scalarOperator[ARRAY] should has the same type
2025-05-14 10:33:52.464+08:00 WARN (starrocks-mysql-nio-pool-23941|1439896) [PruneComplexTypeUtil$Context.checkCanPrune():172] Complex type columnRefOperator[ARRAY<VARCHAR(65533)>] and scalarOperator[ARRAY] should has the same type
2025-05-14 10:33:52.882+08:00 WARN (nioEventLoopGroup-6-35|215) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:34:17.923+08:00 WARN (nioEventLoopGroup-6-50|278) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:34:18.768+08:00 WARN (nioEventLoopGroup-6-58|286) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:34:19.196+08:00 WARN (nioEventLoopGroup-6-3|168) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
2025-05-14 10:34:19.262+08:00 WARN (nioEventLoopGroup-6-9|175) [NodeSelector.seqChooseBackendIds():112] failed to find any backend, needAvailable=true
FE follow节点对应fe.warn.log相关日志:
2025-05-14 11:15:24.139+08:00 WARN (stats-cache-refresher-8|122) [StmtExecutor.executeStmtWithExecPlan():2479] Failed to execute executeStmtWithExecPlan
com.starrocks.common.UserException: Backend node not found. Check if any backend node is down.
at com.starrocks.qe.DefaultCoordinator.getNext(DefaultCoordinator.java:845)
at com.starrocks.qe.StmtExecutor.executeStmtWithExecPlan(StmtExecutor.java:2473)
at com.starrocks.statistic.StatisticExecutor.executeDQL(StatisticExecutor.java:481)
at com.starrocks.statistic.StatisticExecutor.executeStatisticDQL(StatisticExecutor.java:434)
at com.starrocks.statistic.StatisticExecutor.queryColumnStats(StatisticExecutor.java:159)
at com.starrocks.statistic.StatisticExecutor.queryStatisticSync(StatisticExecutor.java:104)
at com.starrocks.sql.optimizer.statistics.ColumnBasicStatsCacheLoader.queryStatisticsData(ColumnBasicStatsCacheLoader.java:129)
at com.starrocks.sql.optimizer.statistics.ColumnBasicStatsCacheLoader.lambda$asyncLoadAll$1(ColumnBasicStatsCacheLoader.java:93)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)