【详述】docker方式部署的存算一体在使用jdbc catalog查询myssql的时候BE频繁重启
【背景】使用docker(image:starrocks/allin1-ubuntu:3.3.0)方式部署的starrocks在创建jdbc外部catalog后，使用show databases、show tables都正常显示，一旦使用select查询外部mysql表的数据，比如select * from table_a,BE就会重启。
【业务影响】 暂无
【是否存算分离】 否
【StarRocks版本】starrocks/allin1-ubuntu:3.3.0
【集群规模】fe与be混部
navicate查询前端截图：
fe.log截图：
2024-09-07 21:37:42.119+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [MetadataMgr$QueryMetadatas.getConnectorMetadata():109] Succeed to register query level connector metadata [catalog:jdbc, queryId: 5b5f18ab-6d1e-11ef-bc40-0242ac1e0002]
2024-09-07 21:37:42.122+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [MetadataMgr.removeQueryMetadata():200] Succeed to deregister query level connector metadata on query id: Optional[5b5f18ab-6d1e-11ef-bc40-0242ac1e0002]
2024-09-07 21:37:42.125+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [MetadataMgr$QueryMetadatas.getConnectorMetadata():109] Succeed to register query level connector metadata [catalog:jdbc, queryId: 5b60030c-6d1e-11ef-bc40-0242ac1e0002]
2024-09-07 21:37:42.147+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [MetadataMgr.removeQueryMetadata():200] Succeed to deregister query level connector metadata on query id: Optional[5b60030c-6d1e-11ef-bc40-0242ac1e0002]
2024-09-07 21:37:42.148+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [QeProcessorImpl.registerQuery():108] register query id = 5b60030c-6d1e-11ef-bc40-0242ac1e0002
2024-09-07 21:37:50.903+08:00 WARN (heartbeat-mgr-pool-0|120) [ThriftRPCRequestExecutor.call():80] Call frontend thrift rpc failed, addr: TNetworkAddress(hostname:127.0.0.1, port:9050), retried: 0
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:172) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:100) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:519) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:387) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:271) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.20.0.jar:0.20.0]
at com.starrocks.thrift.HeartbeatService$Client.recv_heartbeat(HeartbeatService.java:64) ~[starrocks-fe.jar:?]
at com.starrocks.thrift.HeartbeatService$Client.heartbeat(HeartbeatService.java:51) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.lambda$call$0(HeartbeatMgr.java:290) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.call(ThriftRPCRequestExecutor.java:67) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.callNoRetry(ThriftRPCRequestExecutor.java:40) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:287) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:262) ~[starrocks-fe.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:115) ~[?:?]
at java.net.SocketInputStream.read(SocketInputStream.java:168) ~[?:?]
at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:252) ~[?:?]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:292) ~[?:?]
at java.io.BufferedInputStream.read(BufferedInputStream.java:351) ~[?:?]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:170) ~[libthrift-0.20.0.jar:0.20.0]
… 16 more
2024-09-07 21:37:50.906+08:00 WARN (heartbeat-mgr-pool-0|120) [HeartbeatMgr$BackendHeartbeatHandler.call():332] backend heartbeat got exception, addr: 127.0.0.1:9050
org.apache.thrift.transport.TTransportException: java.net.SocketTimeoutException: Read timed out
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:172) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:100) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:519) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:387) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:271) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.20.0.jar:0.20.0]
at com.starrocks.thrift.HeartbeatService$Client.recv_heartbeat(HeartbeatService.java:64) ~[starrocks-fe.jar:?]
at com.starrocks.thrift.HeartbeatService$Client.heartbeat(HeartbeatService.java:51) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.lambda$call$0(HeartbeatMgr.java:290) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.call(ThriftRPCRequestExecutor.java:67) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.callNoRetry(ThriftRPCRequestExecutor.java:40) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:287) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:262) ~[starrocks-fe.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
at java.net.SocketInputStream.socketRead(SocketInputStream.java:115) ~[?:?]
at java.net.SocketInputStream.read(SocketInputStream.java:168) ~[?:?]
at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:252) ~[?:?]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:292) ~[?:?]
at java.io.BufferedInputStream.read(BufferedInputStream.java:351) ~[?:?]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:170) ~[libthrift-0.20.0.jar:0.20.0]
… 16 more
2024-09-07 21:37:50.908+08:00 WARN (heartbeat mgr|15) [HeartbeatMgr.runAfterCatalogReady():170] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.SocketTimeoutException: Read timed out
2024-09-07 21:37:54.981+08:00 INFO (tablet scheduler|34) [ClusterLoadStatistic.classifyBackendByLoad():163] classify backend by load. medium: HDD, avg load score: 0.5, low/mid/high: 0/1/0
2024-09-07 21:37:54.981+08:00 INFO (tablet scheduler|34) [TabletScheduler.updateClusterLoadStatistic():493] update cluster load statistic:
{“beId”:10001,“clusterName”:“default_cluster”,“isAvailable”:true,“cpuCores”:4,“memLimit”:3706474659,“memUsed”:132036512,“mediums”:[{“medium”:“HDD”,“replica”:66,“used”:0,“total”:“143.6GB”,“score”:0.5},{“medium”:“SSD”,“replica”:0,“used”:0,“total”:“0B”,“score”:NaN}],“paths”:[{“beId”:10001,“path”:"/data/deploy/starrocks/be/storage",“pathHash”:-3235560536975967549,“storageMedium”:“HDD”,“total”:154273447936,“used”:0}]}
2024-09-07 21:37:55.128+08:00 WARN (heartbeat-mgr-pool-4|175) [ThriftConnectionPool.reopen():108] reopen error
org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:231) ~[libthrift-0.20.0.jar:0.20.0]
at com.starrocks.rpc.ThriftConnectionPool.reopen(ThriftConnectionPool.java:103) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.call(ThriftRPCRequestExecutor.java:76) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.callNoRetry(ThriftRPCRequestExecutor.java:40) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:287) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:262) ~[starrocks-fe.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]
at java.net.Socket.connect(Socket.java:609) ~[?:?]
at org.apache.thrift.transport.TSocket.open(TSocket.java:226) ~[libthrift-0.20.0.jar:0.20.0]
… 9 more
2024-09-07 21:37:55.129+08:00 WARN (heartbeat-mgr-pool-4|175) [ThriftRPCRequestExecutor.call():80] Call frontend thrift rpc failed, addr: TNetworkAddress(hostname:127.0.0.1, port:9050), retried: 0
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:174) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:100) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:519) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:387) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:271) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.20.0.jar:0.20.0]
at com.starrocks.thrift.HeartbeatService$Client.recv_heartbeat(HeartbeatService.java:64) ~[starrocks-fe.jar:?]
at com.starrocks.thrift.HeartbeatService$Client.heartbeat(HeartbeatService.java:51) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.lambda$call$0(HeartbeatMgr.java:290) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.call(ThriftRPCRequestExecutor.java:67) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.callNoRetry(ThriftRPCRequestExecutor.java:40) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:287) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:262) ~[starrocks-fe.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:186) ~[?:?]
at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:252) ~[?:?]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:292) ~[?:?]
at java.io.BufferedInputStream.read(BufferedInputStream.java:351) ~[?:?]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:170) ~[libthrift-0.20.0.jar:0.20.0]
… 16 more
2024-09-07 21:37:55.131+08:00 WARN (heartbeat-mgr-pool-4|175) [HeartbeatMgr$BackendHeartbeatHandler.call():332] backend heartbeat got exception, addr: 127.0.0.1:9050
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:174) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:100) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:519) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:387) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:271) ~[libthrift-0.20.0.jar:0.20.0]
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.20.0.jar:0.20.0]
at com.starrocks.thrift.HeartbeatService$Client.recv_heartbeat(HeartbeatService.java:64) ~[starrocks-fe.jar:?]
at com.starrocks.thrift.HeartbeatService$Client.heartbeat(HeartbeatService.java:51) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.lambda$call$0(HeartbeatMgr.java:290) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.call(ThriftRPCRequestExecutor.java:67) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.callNoRetry(ThriftRPCRequestExecutor.java:40) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:287) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:262) ~[starrocks-fe.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:186) ~[?:?]
at java.net.SocketInputStream.read(SocketInputStream.java:140) ~[?:?]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:252) ~[?:?]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:292) ~[?:?]
at java.io.BufferedInputStream.read(BufferedInputStream.java:351) ~[?:?]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:170) ~[libthrift-0.20.0.jar:0.20.0]
… 16 more
2024-09-07 21:37:55.131+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [ResultReceiver.getNext():140] fetch result execution exception, finstId=5b60030c-6d1e-11ef-bc40-0242ac1e0003
java.util.concurrent.ExecutionException: A error occurred: errorCode=2001 errorMessage:Channel inactive error!
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy$2.get(ProtobufRpcProxy.java:578) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.starrocks.qe.ResultReceiver.getNext(ResultReceiver.java:98) ~[starrocks-fe.jar:?]
at com.starrocks.qe.DefaultCoordinator.getNext(DefaultCoordinator.java:784) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1146) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:618) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:353) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:548) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:882) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: com.baidu.jprotobuf.pbrpc.ErrorDataException: A error occurred: errorCode=2001 errorMessage:Channel inactive error!
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.doWaitCallback(ProtobufRpcProxy.java:651) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.access$0(ProtobufRpcProxy.java:611) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy$2.get(ProtobufRpcProxy.java:576) ~[jprotobuf-rpc-core-4.2.1.jar:?]
… 11 more
2024-09-07 21:37:55.132+08:00 WARN (heartbeat mgr|15) [HeartbeatMgr.runAfterCatalogReady():170] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.SocketException: Connection reset
2024-09-07 21:37:55.132+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [HostBlacklist.add():91] add black list: 10001, at: 2024-09-07T21:37:55.132578
2024-09-07 21:37:55.132+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [DefaultCoordinator.getNext():787] get next fail, need cancel. status errorCode THRIFT_RPC_ERROR A error occurred: errorCode=2001 errorMessage:Channel inactive error!, query id: 5b60030c-6d1e-11ef-bc40-0242ac1e0002
2024-09-07 21:37:55.132+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [DefaultCoordinator.updateStatus():762] one instance report fail throw updateStatus(), need cancel. job id: -1, query id: 5b60030c-6d1e-11ef-bc40-0242ac1e0002, instance id: NaN
2024-09-07 21:37:55.140+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [BackendServiceClient.cancelPlanFragmentAsync():162] Cancel plan fragment catch a exception, address=127.0.0.1:8060
java.lang.RuntimeException: Unable to validate object
at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:86) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.baidu.jprotobuf.pbrpc.transport.RpcChannel.getConnection(RpcChannel.java:73) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.invoke(ProtobufRpcProxy.java:499) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.sun.proxy.$Proxy35.cancelPlanFragmentAsync(Unknown Source) ~[?:?]
at com.starrocks.rpc.BackendServiceClient.cancelPlanFragmentAsync(BackendServiceClient.java:145) ~[starrocks-fe.jar:?]
at com.starrocks.qe.scheduler.dag.FragmentInstanceExecState.cancelFragmentInstance(FragmentInstanceExecState.java:363) ~[starrocks-fe.jar:?]
at com.starrocks.qe.DefaultCoordinator.cancelRemoteFragmentsAsync(DefaultCoordinator.java:896) ~[starrocks-fe.jar:?]
at com.starrocks.qe.DefaultCoordinator.cancelInternal(DefaultCoordinator.java:882) ~[starrocks-fe.jar:?]
at com.starrocks.qe.DefaultCoordinator.updateStatus(DefaultCoordinator.java:766) ~[starrocks-fe.jar:?]
at com.starrocks.qe.DefaultCoordinator.getNext(DefaultCoordinator.java:790) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1146) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:618) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:353) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:548) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:882) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:506) ~[commons-pool2-2.3.jar:2.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.3.jar:2.3]
at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:80) ~[jprotobuf-rpc-core-4.2.1.jar:?]
… 18 more
2024-09-07 21:37:55.141+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [FragmentInstanceExecState.cancelFragmentInstance():367] cancel plan fragment get a exception, address=127.0.0.1:8060
com.starrocks.rpc.RpcException: Unable to validate object, host: 127.0.0.1
at com.starrocks.rpc.BackendServiceClient.cancelPlanFragmentAsync(BackendServiceClient.java:164) ~[starrocks-fe.jar:?]
at com.starrocks.qe.scheduler.dag.FragmentInstanceExecState.cancelFragmentInstance(FragmentInstanceExecState.java:363) ~[starrocks-fe.jar:?]
at com.starrocks.qe.DefaultCoordinator.cancelRemoteFragmentsAsync(DefaultCoordinator.java:896) ~[starrocks-fe.jar:?]
at com.starrocks.qe.DefaultCoordinator.cancelInternal(DefaultCoordinator.java:882) ~[starrocks-fe.jar:?]
at com.starrocks.qe.DefaultCoordinator.updateStatus(DefaultCoordinator.java:766) ~[starrocks-fe.jar:?]
at com.starrocks.qe.DefaultCoordinator.getNext(DefaultCoordinator.java:790) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1146) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:618) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:353) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:548) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:882) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-09-07 21:37:55.141+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [HostBlacklist.add():91] add black list: 10001, at: 2024-09-07T21:37:55.141835
2024-09-07 21:37:55.142+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [QueryRuntimeProfile.finishAllInstances():239] unfinished instances: [5b60030c-6d1e-11ef-bc40-0242ac1e0003]
2024-09-07 21:37:55.142+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [ExecuteExceptionHandler.handleRpcException():96] Query cancelled by crash of backends or RpcException, [QueryId=5b60030c-6d1e-11ef-bc40-0242ac1e0002] [SQL=SELECT id FROM ods.student limit 1] [Plan=PLAN COST
CPU: 0.0
Memory: 0.0
PLAN FRAGMENT 0(F00)
Output Exprs:id
Input Partition: UNPARTITIONED
RESULT SINK
0:SCAN JDBC
TABLE: student
QUERY: SELECT id FROM student
limit: 1
cardinality: 1
column statistics:
* id–>[-Infinity, Infinity, 0.0, 1.0, 1.0] UNKNOWN
]
com.starrocks.rpc.RpcException: A error occurred: errorCode=2001 errorMessage:Channel inactive error!, host: unknown
at com.starrocks.qe.DefaultCoordinator.getNext(DefaultCoordinator.java:810) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1146) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:618) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:353) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:548) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:882) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-09-07 21:37:55.143+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [StmtExecutor.execute():633] retry 1 times. stmt: SELECT id FROM ods.student limit 1
2024-09-07 21:37:55.143+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [StmtExecutor.execute():661] Query 5b60030c-6d1e-11ef-bc40-0242ac1e0002 failed. Planner profile : Planner:
- – Parser[1] 0
Reason:
2024-09-07 21:37:55.143+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [QueryRuntimeProfile.finishAllInstances():239] unfinished instances: [5b60030c-6d1e-11ef-bc40-0242ac1e0003]
2024-09-07 21:37:55.143+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [QeProcessorImpl.unregisterQuery():149] deregister query id = 5b60030c-6d1e-11ef-bc40-0242ac1e0002
2024-09-07 21:37:55.144+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [StmtExecutor.execute():611] transfer QueryId: 5b60030c-6d1e-11ef-bc40-0242ac1e0002 to 05df3630-4eaf-43b3-a63f-c7e7eb6ecbd9
2024-09-07 21:37:55.145+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [QeProcessorImpl.registerQuery():108] register query id = 05df3630-4eaf-43b3-a63f-c7e7eb6ecbd9
2024-09-07 21:37:55.145+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [StmtExecutor.execute():661] Query 5b60030c-6d1e-11ef-bc40-0242ac1e0002 failed. Planner profile : Planner:
- – Parser[1] 0
Reason:
2024-09-07 21:37:55.145+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [QeProcessorImpl.unregisterQuery():149] deregister query id = 05df3630-4eaf-43b3-a63f-c7e7eb6ecbd9
2024-09-07 21:37:55.145+08:00 INFO (starrocks-mysql-nio-pool-2|1121) [StmtExecutor.execute():742] execute Exception, sql: SELECT id FROM ods.student limit 1, error: Backend node not found. Check if any backend node is down.backend: [127.0.0.1 alive: true inBlacklist: true]
2024-09-07 21:37:55.145+08:00 WARN (starrocks-mysql-nio-pool-2|1121) [DefaultCoordinator.cancel():857] cancel execState of query, this is outside invoke
2024-09-07 21:37:55.917+08:00 WARN (heartbeat-mgr-pool-5|176) [HeartbeatMgr$BackendHeartbeatHandler.call():332] backend heartbeat got exception, addr: 127.0.0.1:9050
org.apache.thrift.TException: java.net.ConnectException: Connection refused (Connection refused)
at com.starrocks.rpc.ThriftRPCRequestExecutor.call(ThriftRPCRequestExecutor.java:60) ~[starrocks-fe.jar:?]
at com.starrocks.rpc.ThriftRPCRequestExecutor.callNoRetry(ThriftRPCRequestExecutor.java:40) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:287) ~[starrocks-fe.jar:?]
at com.starrocks.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:262) ~[starrocks-fe.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
2024-09-07 21:37:55.918+08:00 WARN (heartbeat mgr|15) [HeartbeatMgr.runAfterCatalogReady():170] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused)
2024-09-07 21:37:58.771+08:00 WARN (UpdateBlacklistThread|169) [HostBlacklist.remove():104] remove black list: 10001, at: 2024-09-07T21:37:58.771678
2024-09-07 21:37:58.771+08:00 WARN (UpdateBlacklistThread|169) [HostBlacklist.refresh():207] remove nodeID 10001 from blacklist
2024-09-07 21:37:58.968+08:00 INFO (TaskCleaner|86) [TaskManager.dropTasks():390] drop tasks:[]
2024-09-07 21:37:58.974+08:00 INFO (colocate group clone checker|98) [ColocateTableBalancer.matchGroups():939] finished to match colocate group. cost: 0 ms, in lock time: 0 ms
2024-09-07 21:37:59.321+08:00 INFO (TableKeeper|87) [TableKeeper.correctTable():110] not enough backends in the cluster, expected 3 but got 1
2024-09-07 21:37:59.354+08:00 INFO (leaderCheckpointer|90) [BDBJEJournal.getFinalizedJournalId():272] database names: 9975
2024-09-07 21:37:59.355+08:00 INFO (leaderCheckpointer|90) [Checkpoint.runAfterCatalogReady():107] checkpoint imageVersion 9974, logVersion 0
2024-09-07 21:37:59.365+08:00 INFO (tablet checker|35) [TabletChecker.doCheck():378] finished to check tablets. isUrgent: true, unhealthy/total/added/in_sched/not_ready: 0/0/0/0/0, cost: 0 ms, in lock time: 0 ms, wait time: 0ms
2024-09-07 21:37:59.366+08:00 INFO (tablet checker|35) [TabletChecker.doCheck():378] finished to check tablets. isUrgent: false, unhealthy/total/added/in_sched/not_ready: 0/66/0/0/0, cost: 0 ms, in lock time: 0 ms, wait time: 0ms
be.out截图:
fe