spark connector导入数据异常

spark connector使用集群导入数据报错,数据量50亿条,9个字段, 3个副本,40个分桶,无分区:
spark集群:10台,规格4x16
starrocks集群:10台, 规格4x16
数据导入过程中报错,数据已有一部分入库:
Caused by: java.io.IOException: Failed to commit, partitionId: 58, taskId: 114, epochId: -1

at com.starrocks.connector.spark.sql.write.StarRocksDataWriter.commit(StarRocksDataWriter.java:69)

at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:430)

at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496)

at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:457)

at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:358)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)

at org.apache.spark.scheduler.Task.run(Task.scala:131)

at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)

at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:750)

Caused by: java.lang.RuntimeException: com.starrocks.data.load.stream.exception.StreamLoadFailException: Failed to prepare transaction, please check taskmanager log for details, Transaction{database=‘load_test’, table=‘test’, label=‘e2f78469-56d8-4353-9a58-9a57c4470cd6’, finish=false}

at com.starrocks.data.load.stream.v2.StreamLoadManagerV2.AssertNotException(StreamLoadManagerV2.java:395)

at com.starrocks.data.load.stream.v2.StreamLoadManagerV2.flush(StreamLoadManagerV2.java:327)

at com.starrocks.connector.spark.sql.write.StarRocksDataWriter.commit(StarRocksDataWriter.java:63)

… 12 more

Caused by: com.starrocks.data.load.stream.exception.StreamLoadFailException: Failed to prepare transaction, please check taskmanager log for details, Transaction{database=‘load_test’, table=‘test’, label=‘e2f78469-56d8-4353-9a58-9a57c4470cd6’, finish=false}

at com.starrocks.data.load.stream.v2.TransactionTableRegion.commit(TransactionTableRegion.java:254)

at com.starrocks.data.load.stream.v2.StreamLoadManagerV2.lambda$init$0(StreamLoadManagerV2.java:172)

… 1 more

【StarRocks版本】2.5.5
【集群规模】例如:3fe(1 follower+2observer)+5be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】352787893@qq.com

大佬有解决了吗,碰到同样的问题了

需要拿label=‘e2f78469-56d8-4353-9a58-9a57c4470cd6’到fe的leader节点搜索下具体的报错原因

我也遇到同样的问题,不过我是flink connector,任务可以运行一段时间,然后就会挂掉。拿label在fe的leader节点上也搜索不到相关的信息。

数据有写入吗,没有写入的话,fe leader节点搜不到日志信息的话,需要看下task manager的日志

大哥 最后怎么结局的,我也碰到了