为了更快的定位您的问题,请提供以下信息,谢谢
【详述】查询时fe master出现Thrift Error occurred during processing of message错误,另外两个fe未出现过该错误。
【背景】近期未做操作
【业务影响】查询超时
【是否存算分离】否
【StarRocks版本】例如:2.5.9
【集群规模】例如:3fe + 3be(fe与be混部)
【机器信息】64C/256G/万兆
【联系方式】社区群13-运维菜鸟
【附件】
-
fe.log日志信息===================================
2024-04-11 10:49:30,507 INFO (starrocks-mysql-nio-pool-40604|108374336) [QeProcessorImpl.registerQuery():81] register query id = 1e9d4eab-f7ae-11ee-a34a-6cfe54388a61, job: -1
2024-04-11 10:49:30,509 INFO (starrocks-mysql-nio-pool-40604|108374336) [QeProcessorImpl.unregisterQuery():91] deregister query id 1e9d4eab-f7ae-11ee-a34a-6cfe54388a61
2024-04-11 10:49:30,510 INFO (thrift-server-pool-105722984|108377002) [QeProcessorImpl.reportExecStatus():130] ReportExecStatus() failed, query does not exist, fragment_instance_id=1e9d4eab-f7ae-11ee-a34a-6cfe54388a62, query_id=1e9d4eab-f7ae-11ee-a34a-6cfe54388a61,
2024-04-11 10:49:30,508 ERROR (thrift-server-pool-105722965|108376982) 2024-04-11 10:49:30,508 ERROR (thrift-server-pool-105722965|108376982) [SRTThreadPoolServer$WorkerProcess.run():319] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe (Write failed)
at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:159) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:60) ~[libthrift-0.13.0.jar:0.13.0]
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) ~[libthrift-0.13.0.jar:0.13.0]
at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:311) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_351]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_351]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_351]
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method) ~[?:1.8.0_351]
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) ~[?:1.8.0_351]
at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_351]
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_351]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_351]
at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:157) ~[libthrift-0.13.0.jar:0.13.0]
… 6 more -
be.log日志信息===================================
W0411 10:49:30.462057 68926 utils.cpp:113] Fail to report to master. host=10.3.138.26, port=9020, code=OK
W0411 10:49:30.462103 68926 task_worker_pool.cpp:689] Fail to report workgroup to 10.3.138.26:9020, err=-1
I0411 10:49:30.508556 68999 fragment_executor.cpp:158] Prepare(): query_id=1e9d4eab-f7ae-11ee-a34a-6cfe54388a61 fragment_instance_id=1e9d4eab-f7ae-11ee-a34a-6cfe54388a62 backend_num=0
I0411 10:49:30.510545 67510 pipeline_driver_executor.cpp:224] [Driver] Fail to report exec state due to query not found: fragment_instance_id=1e9d4eab-f7ae-11ee-a34a-6cfe54388a62
I0411 10:49:30.519652 69003 fragment_executor.cpp:158] Prepare(): query_id=1e9deaec-f7ae-11ee-a34a-6cfe54388a61 fragment_instance_id=1e9deaec-f7ae-11ee-a34a-6cfe54388a62 backend_num=1 -
其他说明:
1.服务器CPU、内存、磁盘、网络等资源该时间点正常
2.fe.gc.log正常