某个脚本报跑不了,fe报Connection reset by peer Thrift Error occurred during processing of message

为了更快的定位您的问题,请提供以下信息,谢谢
【详述】问题详细描述
数据说某个脚本跑不了,但是其他的可以跑
【背景】做过哪些操作?
没有
【业务影响】
【StarRocks版本】例如:1.18.2
【集群规模】例如:3fe(1 follower+2observer)+5be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息,请补充下您的联系方式,例如:社区群4-小李或者邮箱,谢谢
【附件】

  • fe.log相应截图
2023-11-06 10:43:51,878 INFO (starrocks-mysql-nio-pool-0|90) [QeProcessorImpl.registerQuery():95] register query id = 51eace31-7c4e-11ee-ac9a-00163e321704
2023-11-06 10:43:51,884 INFO (starrocks-mysql-nio-pool-0|90) [QeProcessorImpl.unregisterQuery():105] deregister query id 51eace31-7c4e-11ee-ac9a-00163e321704
2023-11-06 10:43:51,885 INFO (thrift-server-pool-122|226) [QeProcessorImpl.reportExecStatus():145] ReportExecStatus() failed, query does not exist, fragment_instance_id=51eace31-7c4e-11ee-ac9a-00163e321705, query_id=51eace31-7c4e-11ee-ac9a-00163e321704,
2023-11-06 10:43:51,904 INFO (starrocks-mysql-nio-pool-0|90) [QeProcessorImpl.registerQuery():95] register query id = 51efb032-7c4e-11ee-ac9a-00163e321704
2023-11-06 10:43:51,908 INFO (starrocks-mysql-nio-pool-0|90) [QeProcessorImpl.unregisterQuery():105] deregister query id 51efb032-7c4e-11ee-ac9a-00163e321704
2023-11-06 10:43:51,913 INFO (starrocks-mysql-nio-pool-0|90) [LeaderOpExecutor.forward():177] Forward statement 70 to Leader TNetworkAddress(hostname:10.253.250.93, port:9020)
2023-11-06 10:43:51,934 INFO (replayer|73) [GlobalStateMgr.replayJournalInner():2056] replayed journal from 10289365 - 10289366
2023-11-06 10:43:51,935 INFO (starrocks-mysql-nio-pool-0|90) [LeaderOpExecutor.execute():101] forwarding to master get result max journal id: 10289366
2023-11-06 10:43:51,935 INFO (starrocks-mysql-nio-pool-0|90) [JournalObservable.waitOn():49] waiting for the observer to replay journal to 10289366 with timeout: 3000000 ms
2023-11-06 10:43:52,021 INFO (starrocks-mysql-nio I/O-1|81) [AcceptListener.handleEvent():71] Connection established. remote=/100.97.16.195:40970
2023-11-06 10:43:52,021 WARN (starrocks-mysql-nio-pool-0|90) [AcceptListener.lambda$handleEvent$1():111] connect processor exception because 
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_362]
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_362]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_362]
	at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.8.0_362]
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) ~[?:1.8.0_362]
	at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:153) ~[xnio-nio-3.7.9.Final.jar:3.7.9.Final]
	at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150) ~[xnio-api-3.7.9.Final.jar:3.7.9.Final]
	at org.xnio.channels.Channels.writeBlocking(Channels.java:97) ~[xnio-api-3.7.9.Final.jar:3.7.9.Final]
	at com.starrocks.mysql.nio.NMysqlChannel.realNetSend(NMysqlChannel.java:65) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.MysqlChannel.send(MysqlChannel.java:234) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.MysqlChannel.flush(MysqlChannel.java:257) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.MysqlChannel.sendAndFlush(MysqlChannel.java:325) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.MysqlProto.negotiate(MysqlProto.java:151) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.nio.AcceptListener.lambda$handleEvent$1(AcceptListener.java:86) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]

2023-11-06 10:43:52,277 INFO (starrocks-mysql-nio I/O-2|82) [AcceptListener.handleEvent():71] Connection established. remote=/100.116.66.131:53040
2023-11-06 10:43:52,277 WARN (starrocks-mysql-nio-pool-0|90) [AcceptListener.lambda$handleEvent$1():111] connect processor exception because 
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:1.8.0_362]
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[?:1.8.0_362]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[?:1.8.0_362]
	at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[?:1.8.0_362]
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:470) ~[?:1.8.0_362]
	at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:153) ~[xnio-nio-3.7.9.Final.jar:3.7.9.Final]
	at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150) ~[xnio-api-3.7.9.Final.jar:3.7.9.Final]
	at org.xnio.channels.Channels.writeBlocking(Channels.java:97) ~[xnio-api-3.7.9.Final.jar:3.7.9.Final]
	at com.starrocks.mysql.nio.NMysqlChannel.realNetSend(NMysqlChannel.java:65) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.MysqlChannel.send(MysqlChannel.java:234) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.MysqlChannel.flush(MysqlChannel.java:257) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.MysqlChannel.sendAndFlush(MysqlChannel.java:325) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.MysqlProto.negotiate(MysqlProto.java:151) ~[starrocks-fe.jar:?]
	at com.starrocks.mysql.nio.AcceptListener.lambda$handleEvent$1(AcceptListener.java:86) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
2023-11-06 10:43:52,611 INFO (replayer|73) [DatabaseTransactionMgr.replayUpsertTransactionState():1609] replay a committed transaction TransactionState. txn_id: 913015, label: 9533b8d7-70e9-47ea-b886-657a471e8df8, db id: 11001, table id list: 24511, callback id: -1, coordinator: BE: 10.253.250.91, transaction status: COMMITTED, error replicas num: 0, replica ids: , prepare time: 1699238632575, commit time: 1699238632608, finish time: -1, write cost: 33ms, reason:  attachment: com.starrocks.load.loadv2.ManualLoadTxnCommitAttachment@312c0e65
2023-11-06 10:43:52,611 INFO (replayer|73) [GlobalStateMgr.replayJournalInner():2056] replayed journal from 10289366 - 10289367
2023-11-06 10:43:52,632 INFO (replayer|73) [DatabaseTransactionMgr.replayUpsertTransactionState():1612] replay a visible transaction TransactionState. txn_id: 913015, label: 9533b8d7-70e9-47ea-b886-657a471e8df8, db id: 11001, table id list: 24511, callback id: -1, coordinator: BE: 10.253.250.91, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1699238632575, commit time: 1699238632608, finish time: 1699238632629, write cost: 33ms, publish total cost: 21ms, total cost: 54ms, reason:  attachment: com.starrocks.load.loadv2.ManualLoadTxnCommitAttachment@2d122dc7
2023-11-06 10:43:52,632 INFO (replayer|73) [GlobalStateMgr.replayJournalInner():2056] replayed journal from 10289367 - 10289368
2023-11-06 10:43:52,741 ERROR (thrift-server-pool-124|228) [SRTThreadPoolServer$WorkerProcess.run():319] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:455) ~[libthrift-0.13.0.jar:0.13.0]
	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:354) ~[libthrift-0.13.0.jar:0.13.0]
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:243) ~[libthrift-0.13.0.jar:0.13.0]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:311) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
	at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_362]
	at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_362]
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[?:1.8.0_362]
	at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[?:1.8.0_362]
	at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_362]
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
	... 9 more

  • 慢查询:
    • Profile信息
    • 并行度:show variables like ‘%parallel_fragment_exec_instance_num%’; 1
    • pipeline是否开启:show variables like ‘%pipeline%’;
    • be节点cpu和内存使用率截图
  • 查询报错:
  • be crash
  • 外表查询报错
    • be.out和fe.warn.log

版本fe.out (75.2 KB) StarRocks-3.0.2

脚本是查询脚本么?发下查询语句详细的query_dump 还有fe.gc.log,这个脚本一跑fe就会挂掉么?

刚找到论坛的问题,我先升级3.0.7看看

升级后脚本严格模式有些问题,就先退回3.0.2了,但依然会报这个错,不知道在连什么,业务正常