为了更快的定位您的问题,请提供以下信息,谢谢
【详述】问题详细描述
在测试 routine load 导入kafka数据时发现 某一段时间的数据没有成功导入starRocks,但是后续的数据又成功,routine load依旧是 running状态
W0625 09:57:07.461158 15119 routine_load_task_executor.cpp:505] commit failed id=b9a0fd157ff54555-a27e9e1be0318313, job_id=24446, txn_id: 18118, label=kafka_job_test-24446-b9a0fd15-7ff5-4555-a27e-9e1be0318313, db=kafka_test
W0625 18:09:21.055621 12501 tablet_sink_sender.cpp:296] close channel failed. channel_name=NodeChannel[21646], load_info=load_id=a651d648-5fb5-483e-a786-c8f46410c5ba, txn_id: 31163, parallel=1, compress_type=2, error_msg=[E1008]Reached t
imeout=30000ms @xxx.xxx.xxx.xxx:8060
W0625 18:09:21.066926 12501 plan_fragment_executor.cpp:203] Fail to open fragment, instance_id=a651d648-5fb5-483e-a786-c8f46410c5bb, status=Internal error: [E1008]Reached timeout=30000ms @xxx.xxx.xxx.xxx:8060
/build/starrocks/be/src/exec/tablet_sink_index_channel.cpp:769 _wait_request(closure)
/build/starrocks/be/src/exec/tablet_sink_index_channel.cpp:886 _wait_all_prev_request()
W0625 18:09:21.071269 12501 fragment_mgr.cpp:201] Fail to open fragment a651d648-5fb5-483e-a786-c8f46410c5bb: Internal error: [E1008]Reached timeout=30000ms @xxx.xxx.xxx.xxx:8060
/build/starrocks/be/src/exec/tablet_sink_index_channel.cpp:769 _wait_request(closure)
/build/starrocks/be/src/exec/tablet_sink_index_channel.cpp:886 _wait_all_prev_request()
W0625 18:09:21.071400 12501 stream_load_executor.cpp:111] fragment execute failed, query_id=a651d6485fb5483e-a786c8f46410c5ba, err_msg=[E1008]Reached timeout=30000ms @xxx.xxx.xxx.xxx:8060, id=a651d6485fb5483e-a786c8f46410c5ba, job_id=24449
, txn_id: 31163, label=kafka_job_test-24449-a651d648-5fb5-483e-a786-c8f46410c5ba, db=kafka_test
W0625 18:09:21.071434 17935 routine_load_task_executor.cpp:505] consume failed id=a651d6485fb5483e-a786c8f46410c5ba, job_id=24449, txn_id: 31163, label=kafka_job_test-24449-a651d648-5fb5-483e-a786-c8f46410c5ba, db=kafka_test
【是否存算分离】
否
【StarRocks版本】3.2.4
【集群规模】例如:3fe+3be(fe与be混部)
【表模型】明细表
【导入或者导出方式】 routine load kafka
*************************** 1. row ***************************
Id: 24449
Name: kafka_job_test
CreateTime: 2024-06-25 10:04:56
PauseTime: 2024-06-27 10:47:02
EndTime: NULL
DbName: kafka_test
TableName: role_test
State: PAUSED
DataSourceType: KAFKA
CurrentTaskNum: 0
JobProperties: {“partitions”:"",“partial_update”:“false”,“columnToColumnExpr”:“time”,“maxBatchIntervalS”:“10”,“partial_update_mode”:“null”,“whereExpr”:"",“dataFormat”:“json”,“timezone”:“Asia/Shanghai”,“format”:“json”,“log_rejected_record_num”:“0”,“taskTimeoutSecond”:“60”,“json_root”:"",“maxFilterRatio”:“1.0”,“strict_mode”:“false”,“jsonpaths”:"["$.time"]",“taskConsumeSecond”:“15”,“desireTaskConcurrentNum”:“2”,“maxErrorNum”:“0”,“strip_outer_array”:“false”,“currentTaskConcurrentNum”:“2”,“maxBatchRows”:“200000”}
DataSourceProperties: {“topic”:“a”,“currentKafkaPartitions”:“0,1,2,3”,“brokerList”:“xxx:9092”}
CustomProperties: {“kafka_default_offsets”:“OFFSET_END”,“group.id”:“test”}
Statistic: {“receivedBytes”:75752608,“errorRows”:0,“committedTaskNum”:26100,“loadedRows”:75868,“loadRowsRate”:0,“abortedTaskNum”:10,“totalRows”:75868,“unselectedRows”:0,“receivedBytesRate”:4000,“taskExecuteTimeMs”:15995366}
Progress: {“0”:“15371939”,“1”:“15295942”,“2”:“15350808”,“3”:“15446888”}
TimestampProgress: {“0”:“1719456417485”,“1”:“1719456370498”,“2”:“1719456399424”,“3”:“1719456363436”}
ReasonOfStateChanged: ErrorReason{errCode = 100, msg=‘User root pauses routine load job’}
ErrorLogUrls:
TrackingSQL:
OtherMsg: 2024-06-27 09:04:31: failed to send task: java.net.SocketException: Connection timed out (Write failed)
LatestSourcePosition: {“0”:“15371940”,“1”:“15295943”,“2”:“15350809”,“3”:“15446889”}
异常时间时 06-25 10:00:00 前后5分钟
补充一个 be.INFO 的信息
I0625 09:57:07.472793 15119 routine_load_task_executor.cpp:339] finished routine load task id=b9a0fd157ff54555-a27e9e1be0318313, job_id=24446, txn_id: 18118, label=kafka_job_test-24446-b9a0fd15-7ff5-4555-a27e-9e1be0318313, db=kafka_test,
status: fail to execute commit task: txn 18118 could not be COMMITTED while task kafka_job_test-24446-b9a0fd15-7ff5-4555-a27e-9e1be0318313 has been aborted., current tasks num: 0