为了更快的定位您的问题,请提供以下信息,谢谢
【详述】使用spark connector 向 sr 写数据
【背景】做过哪些操作?
【业务影响】
【是否存算分离】是
【StarRocks版本】3.2.1
【集群规模】1 fe + 1 be(fe与be混部)
【机器信息】8C/64G
客户端日志
WARN [task-result-getter-3] TaskSetManager: Lost task 3.0 in stage 2.0 (TID 63) (emr-worker-1.cluster-325063 executor 2): java.io.IOException: Failed to commit, partitionId: 3, taskId: 63, epochId: -1
at com.starrocks.connector.spark.sql.write.StarRocksDataWriter.commit(StarRocksDataWriter.java:96)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:430)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1496)
at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:457)
at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:358)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:131)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Caused by: java.lang.RuntimeException: com.starrocks.data.load.stream.exception.StreamLoadFailException: Stream load failed because of unknown exception, db: rta, table: ads_user_center_with_qyid_new, label: spark-792fad0e-336b-4eb1-b4ad-8e59b7285de2
BE warning 日志
W0115 10:55:12.507268 29931 stream_load_executor.cpp:187] begin transaction failed, errmsg=Label [spark-792fad0e-336b-4eb1-b4ad-8e59b7285de2] has already been used.id=8d4dc61029805d0e-0e1251182a0ff393, job_id=-1, txn_id: -1, label=spark-792fad0e-336b-4eb1-b4ad-8e59b7285de2, db=rta
W0115 10:55:12.508262 29930 stream_load_executor.cpp:187] begin transaction failed, errmsg=Label [spark-a546a53d-cc33-476e-b54b-f23dc9690478] has already been used.id=d64b8305992c5ce6-0206f720e9d3e292, job_id=-1, txn_id: -1, label=spark-a546a53d-cc33-476e-b54b-f23dc9690478, db=rta
W0115 10:55:12.537443 29894 stream_load_executor.cpp:187] begin transaction failed, errmsg=Label [spark-f39a7512-ac3b-4788-b253-573c4877f8f3] has already been used.id=6644333f98055fbc-8611cec35a96e68b, job_id=-1, txn_id: -1, label=spark-f39a7512-ac3b-4788-b253-573c4877f8f3, db=rta
W0115 10:55:12.544826 29907 stream_load_executor.cpp:187] begin transaction failed, errmsg=Label [spark-f88ac2ed-0cc6-4816-960b-4b48c5a94b87] has already been used.id=34471df15572c1c6-4bf2dae31a67f1a5, job_id=-1, txn_id: -1, label=spark-f88ac2ed-0cc6-4816-960b-4b48c5a94b87, db=rta
W0115 10:55:12.563730 29929 stream_load_executor.cpp:187] begin transaction failed, errmsg=Label [spark-ee2b2415-25ce-49ba-9287-309d37d911fd] has already been used.id=d0493f85869a5c1a-1176148b965e3bb1, job_id=-1, txn_id: -1, label=spark-ee2b2415-25ce-49ba-9287-309d37d911fd, db=rta
W0115 10:55:22.526269 29924 stream_load_executor.cpp:187] begin transaction failed, errmsg=Label [spark-792fad0e-336b-4eb1-b4ad-8e59b7285de2] has already been used.id=a54a3a8dd4c91e97-101165b057a35095, job_id=-1, txn_id: -1, label=spark-792fad0e-336b-4eb1-b4ad-8e59b7285de2, db=rta
be.out 日志
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/StarRocks/be/lib/jni-packages/starrocks-jdbc-bridge-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/StarRocks/be/lib/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
start time: Mon Jan 15 11:17:01 CST 2024
start time: Mon Jan 15 11:26:01 CST 2024