查询外部表be瞬间全部挂掉

💬 StarRocks 用户问答 日常运维
#1

【详述】问题详细描述
【背景】做过哪些操作？
【业务影响】
【StarRocks版本】例如：1.18.2
【集群规模】例如：3fe（1 follower+2observer）+5be（fe与be混部）
【机器信息】CPU虚拟核/内存/网卡，例如：32C/64G/万兆
【附件】

  • fe.log/beINFO/相应截图
  • 慢查询：
#2

请问下查询了什么外表么？有没有什么比较特殊的类型？比如array

#3

hive外部表，通过array_contains(u_actioninfo_008,‘M7500’)过滤array字段

#4

可以尝试下不查询array看下有没有问题

#5

查array没问题，过滤瞬间be全挂

#6

辛苦补充下 master fe.log日志 以及be.INFO 的日志。

#7

咱们starrocks是什么版本的呢?

#8

starrocks 2.4.0

#9

2022-11-22 17:44:03,743 WARN (starrocks-mysql-nio-pool-533|89181) [StmtExecutor.execute():448] retry 1 times. stmt: select * from test_db.edw_ic_adm_dg_c_starrocks_operation_prefer_material_std_df where array_con
tains(u_actioninfo_008,‘M7500’) > 1 limit 10
2022-11-22 17:44:03,743 INFO (starrocks-mysql-nio-pool-533|89181) [QeProcessorImpl.unregisterQuery():79] deregister query id 31d9455c-6a4a-11ed-93fc-e2bfd878f4b4
2022-11-22 17:44:03,743 INFO (starrocks-mysql-nio-pool-533|89181) [StmtExecutor.execute():423] transfer QueryId: 31d9455c-6a4a-11ed-93fc-e2bfd878f4b4 to 3ed7b6f8-d5e5-4dfa-b9b7-e196183f3bff
2022-11-22 17:44:03,744 INFO (starrocks-mysql-nio-pool-533|89181) [QeProcessorImpl.registerQuery():69] register query id = 3ed7b6f8-d5e5-4dfa-b9b7-e196183f3bff, job: -1
2022-11-22 17:44:03,744 INFO (thrift-server-pool-480|89164) [QeProcessorImpl.reportExecStatus():118] ReportExecStatus() failed, query does not exist, fragment_instance_id=31d9455c-6a4a-11ed-93fc-e2bfd878f4b5, que
ry_id=31d9455c-6a4a-11ed-93fc-e2bfd878f4b4,
2022-11-22 17:44:03,755 ERROR (thrift-server-pool-475|89159) [SRTThreadPoolServer$WorkerProcess.run():318] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:310) [starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_251]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_251]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_251]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
… 9 more
2022-11-22 17:44:03,755 ERROR (thrift-server-pool-477|89161) [SRTThreadPoolServer$WorkerProcess.run():318] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:310) [starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_251]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_251]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_251]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
… 9 more
2022-11-22 17:44:03,755 ERROR (thrift-server-pool-479|89163) [SRTThreadPoolServer$WorkerProcess.run():318] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:310) [starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_251]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_251]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_251]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
… 9 more
2022-11-22 17:44:03,755 ERROR (thrift-server-pool-468|89151) [SRTThreadPoolServer$WorkerProcess.run():318] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:310) [starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_251]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_251]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_251]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
… 9 more
2022-11-22 17:44:03,756 INFO (thrift-server-pool-477|89161) [ThriftServerEventProcessor.deleteContext():112] delete thrift context. client: TNetworkAddress(hostname:10.24.2.32, port:53706)
2022-11-22 17:44:03,756 INFO (thrift-server-pool-475|89159) [ThriftServerEventProcessor.deleteContext():112] delete thrift context. client: TNetworkAddress(hostname:10.24.2.32, port:53720)
2022-11-22 17:44:03,756 INFO (thrift-server-pool-479|89163) [ThriftServerEventProcessor.deleteContext():112] delete thrift context. client: TNetworkAddress(hostname:10.24.2.32, port:53704)
2022-11-22 17:44:03,756 INFO (thrift-server-pool-468|89151) [ThriftServerEventProcessor.deleteContext():112] delete thrift context. client: TNetworkAddress(hostname:10.24.2.32, port:53722)
2022-11-22 17:44:03,756 ERROR (thrift-server-pool-484|89172) [SRTThreadPoolServer$WorkerProcess.run():318] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:310) [starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_251]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_251]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_251]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
… 9 more
2022-11-22 17:44:03,756 INFO (thrift-server-pool-484|89172) [ThriftServerEventProcessor.deleteContext():112] delete thrift context. client: TNetworkAddress(hostname:10.24.2.32, port:53824)
2022-11-22 17:44:03,782 WARN (starrocks-mysql-nio-pool-533|89181) [BackendServiceClient.execBatchPlanFragmentsAsync():111] Execute batch plan fragments catch a exception, address=10.24.2.32:8060
java.lang.RuntimeException: Unable to validate object
at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:86) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.baidu.jprotobuf.pbrpc.transport.RpcChannel.getConnection(RpcChannel.java:73) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.invoke(ProtobufRpcProxy.java:499) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.sun.proxy.$Proxy34.execBatchPlanFragmentsAsync(Unknown Source) ~[?:?]
at com.starrocks.rpc.BackendServiceClient.execBatchPlanFragmentsAsync(BackendServiceClient.java:97) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator$BackendExecState.execRemoteBatchFragmentsAsync(Coordinator.java:2721) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator.deliverExecBatchFragmentsRequests(Coordinator.java:977) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator.deliverExecFragments(Coordinator.java:610) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator.exec(Coordinator.java:516) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:758) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:431) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:319) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:434) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:680) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:506) ~[commons-pool2-2.3.jar:2.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.3.jar:2.3]
at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:80) ~[jprotobuf-rpc-core-4.2.1.jar:?]
… 17 more
2022-11-22 17:44:03,932 INFO (starrocks-mysql-nio I/O-4|109) [AcceptListener.handleEvent():54] Connection established. remote=/10.24.2.98:5739
2022-11-22 17:44:03,933 INFO (starrocks-mysql-nio-pool-537|89201) [ComputeNodeProcDir.getClusterComputeNodesInfos():109] compute node proc get tablet num cost: 0, total cost: 0
2022-11-22 17:44:03,937 ERROR (thrift-server-pool-471|89154) [SRTThreadPoolServer$WorkerProcess.run():318] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:310) [starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_251]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_251]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_251]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
… 9 more
2022-11-22 17:44:03,937 ERROR (thrift-server-pool-467|89150) [SRTThreadPoolServer$WorkerProcess.run():318] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:310) [starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_251]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_251]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_251]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
… 9 more
2022-11-22 17:44:03,937 INFO (thrift-server-pool-467|89150) [ThriftServerEventProcessor.deleteContext():112] delete thrift context. client: TNetworkAddress(hostname:10.24.2.30, port:46828)
2022-11-22 17:44:03,937 INFO (thrift-server-pool-471|89154) [ThriftServerEventProcessor.deleteContext():112] delete thrift context. client: TNetworkAddress(hostname:10.24.2.30, port:44562)
2022-11-22 17:44:03,937 ERROR (thrift-server-pool-478|89162) [SRTThreadPoolServer$WorkerProcess.run():318] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:310) [starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_251]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_251]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_251]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
… 9 more
2022-11-22 17:44:03,937 ERROR (thrift-server-pool-486|89174) [SRTThreadPoolServer$WorkerProcess.run():318] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:310) [starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_251]
at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_251]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_251]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_251]
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
… 9 more
2022-11-22 17:44:03,938 INFO (thrift-server-pool-478|89162) [ThriftServerEventProcessor.deleteContext():112] delete thrift context. client: TNetworkAddress(hostname:10.24.2.30, port:46752)
2022-11-22 17:44:03,938 INFO (thrift-server-pool-486|89174) [ThriftServerEventProcessor.deleteContext():112] delete thrift context. client: TNetworkAddress(hostname:10.24.2.30, port:52478)
2022-11-22 17:44:03,938 WARN (starrocks-mysql-nio-pool-533|89181) [Coordinator.deliverExecBatchFragmentsRequests():991] catch a execute exception
java.util.concurrent.ExecutionException: A error occurred: errorCode=2001 errorMessage:Connection reset by peer
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy$2.get(ProtobufRpcProxy.java:578) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.starrocks.qe.Coordinator.deliverExecBatchFragmentsRequests(Coordinator.java:985) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator.deliverExecFragments(Coordinator.java:610) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator.exec(Coordinator.java:516) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:758) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:431) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:319) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:434) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:680) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: com.baidu.jprotobuf.pbrpc.ErrorDataException: A error occurred: errorCode=2001 errorMessage:Connection reset by peer
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.doWaitCallback(ProtobufRpcProxy.java:651) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.access$0(ProtobufRpcProxy.java:611) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy$2.get(ProtobufRpcProxy.java:576) ~[jprotobuf-rpc-core-4.2.1.jar:?]
… 12 more
2022-11-22 17:44:03,938 WARN (starrocks-mysql-nio-pool-533|89181) [Coordinator.deliverExecBatchFragmentsRequests():1006] exec plan fragment failed, errmsg=exec rpc error. backend id: 10003, code: THRIFT_RPC_ERROR
, fragmentId=F00, backend=10.24.2.30:9060
2022-11-22 17:44:03,941 WARN (starrocks-mysql-nio-pool-533|89181) [BackendServiceClient.cancelPlanFragmentAsync():154] Cancel plan fragment catch a exception, address=10.24.2.30:8060
java.lang.RuntimeException: Unable to validate object
at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:86) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.baidu.jprotobuf.pbrpc.transport.RpcChannel.getConnection(RpcChannel.java:73) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.invoke(ProtobufRpcProxy.java:499) ~[jprotobuf-rpc-core-4.2.1.jar:?]
at com.sun.proxy.$Proxy34.cancelPlanFragmentAsync(Unknown Source) ~[?:?]
at com.starrocks.rpc.BackendServiceClient.cancelPlanFragmentAsync(BackendServiceClient.java:137) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator$BackendExecState.cancelFragmentInstance(Coordinator.java:2629) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator.cancelRemoteFragmentsAsync(Coordinator.java:1347) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator.cancelInternal(Coordinator.java:1336) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator.deliverExecBatchFragmentsRequests(Coordinator.java:1009) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator.deliverExecFragments(Coordinator.java:610) ~[starrocks-fe.jar:?]
at com.starrocks.qe.Coordinator.exec(Coordinator.java:516) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:758) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:431) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:319) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:434) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:680) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_251]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_251]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_251]
Caused by: java.util.NoSuchElementException: Unable to validate object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:506) ~[commons-pool2-2.3.jar:2.3]
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363) ~[commons-pool2-2.3.jar:2.3]
at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:80) ~[jprotobuf-rpc-core-4.2.1.jar:?]
… 19 more

#10

be.INFO 没有异常时间点的日志

#11

您好，可以提供下schema还有sql么？

#12

±-----------------±--------------------±-----±-----±--------±------+
| Field | Type | Null | Key | Default | Extra |
±-----------------±--------------------±-----±-----±--------±------+
| user_id | VARCHAR(200) | Yes | true | NULL | |
| u_actioninfo_001 | INT | Yes | true | NULL | |
| u_actioninfo_002 | INT | Yes | true | NULL | |
| u_actioninfo_003 | INT | Yes | true | NULL | |
| u_actioninfo_004 | INT | Yes | true | NULL | |
| u_actioninfo_007 | ARRAY<VARCHAR(200)> | Yes | true | NULL | |
| u_actioninfo_008 | ARRAY<VARCHAR(200)> | Yes | true | NULL | |
| u_actioninfo_009 | ARRAY<VARCHAR(200)> | Yes | true | NULL | |
| u_actioninfo_010 | ARRAY<VARCHAR(200)> | Yes | true | NULL | |
| u_actioninfo_011 | ARRAY<VARCHAR(200)> | Yes | true | NULL | |
| u_actioninfo_012 | ARRAY<VARCHAR(200)> | Yes | true | NULL | |
| u_actioninfo_013 | ARRAY<VARCHAR(200)> | Yes | true | NULL | |
| u_actioninfo_014 | VARCHAR(200) | Yes | true | NULL | |
| u_actioninfo_016 | VARCHAR(200) | Yes | true | NULL | |
| u_actioninfo_018 | ARRAY<VARCHAR(200)> | Yes | true | NULL | |
| u_actioninfo_020 | VARCHAR(200) | Yes | true | NULL | |
| u_actioninfo_021 | INT | Yes | true | NULL | |
| u_actioninfo_022 | VARCHAR(200) | Yes | true | NULL | |
| u_actioninfo_025 | INT | Yes | true | NULL | |
| u_actioninfo_026 | VARCHAR(200) | Yes | true | NULL | |
| u_actioninfo_028 | VARCHAR(200) | Yes | true | NULL | |
| u_actioninfo_029 | INT | Yes | true | NULL | |
| u_actioninfo_039 | VARCHAR(200) | Yes | true | NULL | |
| u_assetinfo_002 | INT | Yes | true | NULL | |
| u_assetinfo_003 | INT | Yes | true | NULL | |
| u_assetinfo_004 | INT | Yes | true | NULL | |
| u_assetinfo_005 | INT | Yes | true | NULL | |
| u_assetinfo_007 | INT | Yes | true | NULL | |
| u_businfo_001 | INT | Yes | true | NULL | |
| u_businfo_002 | INT | Yes | true | NULL | |
| u_businfo_003 | INT | Yes | true | NULL | |
| u_businfo_004 | INT | Yes | true | NULL | |
| u_businfo_005 | INT | Yes | true | NULL | |
| u_businfo_006 | INT | Yes | true | NULL | |
| u_businfo_007 | INT | Yes | true | NULL | |
| u_businfo_008 | INT | Yes | true | NULL | |
| u_businfo_015 | BIGINT | Yes | true | NULL | |
| u_businfo_040 | BIGINT | Yes | true | NULL | |
| u_businfo_041 | BIGINT | Yes | true | NULL | |
| u_businfo_042 | BIGINT | Yes | true | NULL | |
| u_businfo_043 | BIGINT | Yes | true | NULL | |
| u_businfo_044 | BIGINT | Yes | true | NULL | |
| u_businfo_045 | DOUBLE | Yes | true | NULL | |
| u_businfo_052 | DOUBLE | Yes | true | NULL | |
| u_businfo_053 | DOUBLE | Yes | true | NULL | |
| u_businfo_054 | DOUBLE | Yes | true | NULL | |
| u_deviceinfo_002 | VARCHAR(200) | Yes | true | NULL | |
| u_deviceinfo_003 | VARCHAR(200) | Yes | true | NULL | |
| u_deviceinfo_004 | VARCHAR(200) | Yes | true | NULL | |
| u_userinfo_001 | VARCHAR(200) | Yes | true | NULL | |
| u_userinfo_008 | VARCHAR(200) | Yes | true | NULL | |
| platform | VARCHAR(200) | Yes | true | NULL | |
| registration_id | VARCHAR(200) | Yes | true | NULL | |
| eid | VARCHAR(200) | Yes | true | NULL | |
| ename | VARCHAR(200) | Yes | true | NULL | |
| row | DECIMAL(36,15) | Yes | true | NULL | |
| data_type | VARCHAR(200) | Yes | true | NULL | |
| total_case_cnt | BIGINT | Yes | true | NULL | |
| case_type | VARCHAR(200) | Yes | true | NULL | |
| case_cnt | BIGINT | Yes | true | NULL | |
| material_score | FLOAT | Yes | true | NULL | |
±-----------------±--------------------±-----±-----±--------±------+

select * from test_db.edw_ic_adm_dg_c_starrocks_operation_prefer_material_std_df where array_con
tains(u_actioninfo_008,‘M7500’) > 0

#13

辛苦您执行下show create table table_name;然后发下吧

#14

CREATE EXTERNAL TABLE edw_ic_adm_dg_c_starrocks_operation_prefer_material_std_df (
user_id varchar(200) NULL COMMENT “”,
u_actioninfo_001 int(11) NULL COMMENT “”,
u_actioninfo_002 int(11) NULL COMMENT “”,
u_actioninfo_003 int(11) NULL COMMENT “”,
u_actioninfo_004 int(11) NULL COMMENT “”,
u_actioninfo_007 ARRAY<varchar(200)> NULL COMMENT “”,
u_actioninfo_008 ARRAY<varchar(200)> NULL COMMENT “”,
u_actioninfo_009 ARRAY<varchar(200)> NULL COMMENT “”,
u_actioninfo_010 ARRAY<varchar(200)> NULL COMMENT “”,
u_actioninfo_011 ARRAY<varchar(200)> NULL COMMENT “”,
u_actioninfo_012 ARRAY<varchar(200)> NULL COMMENT “”,
u_actioninfo_013 ARRAY<varchar(200)> NULL COMMENT “”,
u_actioninfo_014 varchar(200) NULL COMMENT “”,
u_actioninfo_016 varchar(200) NULL COMMENT “”,
u_actioninfo_018 ARRAY<varchar(200)> NULL COMMENT “”,
u_actioninfo_020 varchar(200) NULL COMMENT “”,
u_actioninfo_021 int(11) NULL COMMENT “”,
u_actioninfo_022 varchar(200) NULL COMMENT “”,
u_actioninfo_025 int(11) NULL COMMENT “”,
u_actioninfo_026 varchar(200) NULL COMMENT “”,
u_actioninfo_028 varchar(200) NULL COMMENT “”,
u_actioninfo_029 int(11) NULL COMMENT “”,
u_actioninfo_039 varchar(200) NULL COMMENT “”,
u_assetinfo_002 int(11) NULL COMMENT “”,
u_assetinfo_003 int(11) NULL COMMENT “”,
u_assetinfo_004 int(11) NULL COMMENT “”,
u_assetinfo_005 int(11) NULL COMMENT “”,
u_assetinfo_007 int(11) NULL COMMENT “”,
u_businfo_001 int(11) NULL COMMENT “”,
u_businfo_002 int(11) NULL COMMENT “”,
u_businfo_003 int(11) NULL COMMENT “”,
u_businfo_004 int(11) NULL COMMENT “”,
u_businfo_005 int(11) NULL COMMENT “”,
u_businfo_006 int(11) NULL COMMENT “”,
u_businfo_007 int(11) NULL COMMENT “”,
u_businfo_008 int(11) NULL COMMENT “”,
u_businfo_015 bigint(20) NULL COMMENT “”,
u_businfo_040 bigint(20) NULL COMMENT “”,
u_businfo_041 bigint(20) NULL COMMENT “”,
u_businfo_042 bigint(20) NULL COMMENT “”,
u_businfo_043 bigint(20) NULL COMMENT “”,
u_businfo_044 bigint(20) NULL COMMENT “”,
u_businfo_045 double NULL COMMENT “”,
u_businfo_052 double NULL COMMENT “”,
u_businfo_053 double NULL COMMENT “”,
u_businfo_054 double NULL COMMENT “”,
u_deviceinfo_002 varchar(200) NULL COMMENT “”,
u_deviceinfo_003 varchar(200) NULL COMMENT “”,
u_deviceinfo_004 varchar(200) NULL COMMENT “”,
u_userinfo_001 varchar(200) NULL COMMENT “”,
u_userinfo_008 varchar(200) NULL COMMENT “”,
platform varchar(200) NULL COMMENT “”,
registration_id varchar(200) NULL COMMENT “”,
eid varchar(200) NULL COMMENT “”,
ename varchar(200) NULL COMMENT “”,
row decimal128(36, 15) NULL COMMENT “”,
data_type varchar(200) NULL COMMENT “”,
total_case_cnt bigint(20) NULL COMMENT “”,
case_type varchar(200) NULL COMMENT “”,
case_cnt bigint(20) NULL COMMENT “”,
material_score float NULL COMMENT “”
) ENGINE=HIVE
COMMENT “PARTITION BY ()”
PROPERTIES (
“database” = “edw_ic”,
“table” = “adm_dg_c_starrocks_operation_prefer_material_std_df”,
“resource” = “hive_101”,
“hive.metastore.uris” = “thrift://xxx:9083”
);

#15

什么版本？可以 select current_version() 看看。

1.18.2 这个版本非常老了，是没有pipeline这个特性的。

#16

image

#17

您好，我这里没有办法复现这个问题。

你能可以先升级到最新的2.4.1看看这个问题是否已经被修复吗？

另外你们最近是否有做升级schema的操作呢?不仅限于array字段，其他字段是否做过schema change.

然后我觉得可以测试下面几个query

  1. 只选择array字段看看是否出现问题

select u_actioninfo_008 from test_db.edw_ic_adm_dg_c_starrocks_operation_prefer_material_std_df where array_con
tains(u_actioninfo_008,‘M7500’) > 0

  1. 选择array字段+user_id(varchar)字段看看是否还是会出现问题

select user_id from test_db.edw_ic_adm_dg_c_starrocks_operation_prefer_material_std_df where array_con
tains(u_actioninfo_008,‘M7500’) > 0

#18

升级了 没得用，还是瞬间全挂

#19

你们最近是否有做升级schema的操作呢?不仅限于array字段，其他字段是否做过schema change？

然后我觉得可以测试下面几个query

  1. 只选择array字段看看是否出现问题

select u_actioninfo_008 from test_db.edw_ic_adm_dg_c_starrocks_operation_prefer_material_std_df where array_con
tains(u_actioninfo_008,‘M7500’) > 0

  1. 选择array字段+user_id(varchar)字段看看是否还是会出现问题

select user_id from test_db.edw_ic_adm_dg_c_starrocks_operation_prefer_material_std_df where array_con
tains(u_actioninfo_008,‘M7500’) > 0

#20

没有，表删了重建 也不行

京ICP备2022026421号-1