CloudCanal数据导入表结构同步成功，数据没有同步

💬 StarRocks 用户问答
#1

【详述】CloudCanal数据导入表结构同步成功，但是数据没有同步，就像是卡住了一样。日志没有报错
【背景】CloudCanal增量数据同步

image
image1408×571 39 KB

image
image1621×546 42.9 KB

image
image1128×588 49.3 KB

部分日志：

2022-01-18 15:10:57.266 [task-position-report-4-thd-0] INFO rsocket - [task->sidecar],route name:taskPositionReport.request id:c865834f-782d-11ec-b040-a14972654227,elapse:6 ms.
2022-01-18 15:10:57.267 [task-position-report-4-thd-0] INFO c.c.c.base.service.task.position.PositionReporterImpl - report task (11) position successfully
2022-01-18 15:11:01.858 [reactor-tcp-epoll-1] INFO c.c.cloudcanal.task.rsocket.impl.AsyncReceiverImpl - receive async result. request id:cb205d90-782d-11ec-b040-cffdff5bc17f
2022-01-18 15:11:01.858 [heartbeat-report-3-thd-0] INFO rsocket - [task->sidecar],route name:taskReceiveHeartBeat.request id:cb205d90-782d-11ec-b040-cffdff5bc17f,elapse:18 ms.
2022-01-18 15:11:06.860 [stats-report-2-thd-0] INFO c.c.c.base.service.task.monitor.TaskMonitorManager - report task stats(15) to sidecar.
2022-01-18 15:11:06.863 [reactor-tcp-epoll-1] INFO c.c.cloudcanal.task.rsocket.impl.AsyncReceiverImpl - receive async result. request id:ce1e5b51-782d-11ec-b040-cbe046701093
2022-01-18 15:11:06.863 [stats-report-2-thd-0] INFO rsocket - [task->sidecar],route name:taskMappedStatReport.request id:ce1e5b51-782d-11ec-b040-cbe046701093,elapse:3 ms.
2022-01-18 15:11:07.277 [reactor-tcp-epoll-1] INFO c.c.cloudcanal.task.rsocket.impl.AsyncReceiverImpl - receive async result. request id:ce5ceaf2-782d-11ec-b040-953683cbe9f1
2022-01-18 15:11:07.277 [task-position-report-4-thd-0] INFO rsocket - [task->sidecar],route name:taskPositionReport.request id:ce5ceaf2-782d-11ec-b040-953683cbe9f1,elapse:7 ms.
2022-01-18 15:11:07.277 [task-position-report-4-thd-0] INFO c.c.c.base.service.task.position.PositionReporterImpl - report task (11) position successfully
2022-01-18 15:11:07.804 [reactor-tcp-epoll-1] INFO c.c.cloudcanal.task.rsocket.impl.AsyncReceiverImpl - receive async result. request id:ceada303-782d-11ec-b040-3f2cb88f4f91
2022-01-18 15:11:07.804 [task-config-refresh-7-thd-0] INFO rsocket - [task->sidecar],route name:taskConfigFetch.request id:ceada303-782d-11ec-b040-3f2cb88f4f91,elapse:5 ms.
2022-01-18 15:11:07.817 [task-config-refresh-7-thd-0] INFO c.clougence.cloudcanal.task.util.CloudCanalConfigHelper - get task (canal2236jt07z4a_FULL) config successfully.
2022-01-18 15:11:11.870 [reactor-tcp-epoll-1] INFO c.c.cloudcanal.task.rsocket.impl.AsyncReceiverImpl - receive async result. request id:d118fdb4-782d-11ec-b040-59104b9524fb
2022-01-18 15:11:11.870 [heartbeat-report-3-thd-0] INFO rsocket - [task->sidecar],route name:taskReceiveHeartBeat.request id:d118fdb4-782d-11ec-b040-59104b9524fb,elapse:12 ms.
2022-01-18 15:11:16.864 [stats-report-2-thd-0] INFO c.c.c.base.service.task.monitor.TaskMonitorManager - report task stats(15) to sidecar.

#2

@U_1638258806011_0925 您好，看你的截图是完成结构迁移后，全量阶段没有正常启动。可以按照如下方式看下异常监控日志，有没有产生一些异常信息？

image
image3178×692 200 KB

#3

这个异常是什么原因呢

com.clougence.cloudcanal.base.service.task.exception.runtime.ApplierHandlerException: apply error,msg:RuntimeException: Failed to flush data to StarRocks, Error response:
{“Status”:“Fail”,“BeginTxnTimeMs”:0,“Message”:“close index channel failed, load_id=5242784f-f218-e1a6-95a2-45cebf1cdfb8”,“NumberUnselectedRows”:0,“CommitAndPublishTimeMs”:0,“Label”:“5c5ab558-f26a-4210-a22e-6e4bb7a90be7”,“LoadBytes”:27392,“StreamLoadPutTimeMs”:0,“NumberTotalRows”:0,“WriteDataTimeMs”:14,“TxnId”:1950071,“LoadTimeMs”:15,“ReadDataTimeMs”:0,“NumberLoadedRows”:0,“NumberFilteredRows”:0}
{}

at com.clougence.cloudcanal.starrocks.worker.writer.full.StarRocksFullApplyHandler.handle(StarRocksFullApplyHandler.java:62)
at com.clougence.cloudcanal.starrocks.worker.writer.full.StarRocksFullApplyHandler.handle(StarRocksFullApplyHandler.java:22)
at com.clougence.cloudcanal.task.applier.full.FullApplyWorkHandler.onEvent(FullApplyWorkHandler.java:70)
at com.clougence.cloudcanal.task.applier.full.FullApplyWorkHandler.onEvent(FullApplyWorkHandler.java:21)
at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to flush data to StarRocks, Error response:
{“Status”:“Fail”,“BeginTxnTimeMs”:0,“Message”:“close index channel failed, load_id=5242784f-f218-e1a6-95a2-45cebf1cdfb8”,“NumberUnselectedRows”:0,“CommitAndPublishTimeMs”:0,“Label”:“5c5ab558-f26a-4210-a22e-6e4bb7a90be7”,“LoadBytes”:27392,“StreamLoadPutTimeMs”:0,“NumberTotalRows”:0,“WriteDataTimeMs”:14,“TxnId”:1950071,“LoadTimeMs”:15,“ReadDataTimeMs”:0,“NumberLoadedRows”:0,“NumberFilteredRows”:0}
{}

at com.clougence.cloudcanal.starrocks.worker.writer.executor.full.StarRocksFullStreamLoadExecutor.insert(StarRocksFullStreamLoadExecutor.java:74)
at com.clougence.cloudcanal.starrocks.worker.writer.full.StarRocksFullApplyHandler.handle(StarRocksFullApplyHandler.java:48)
… 5 more

#4

close index channel failed，这个报错是因为导入频率较快，可以设置下导入的batch size或者增加停顿时间来避免这个报错的频繁出现。

可以参考下https://www.askcug.com/topic 这个文档中的FAQ。

全量和增量的参数调整可以通过任务详情页面打开任务参数.
全量是，fullRingBufferSize和fullBatchSize
增量是，increRingBufferSize和increBatchSize

例如开4G规格，全量的batch size开4K以上应该问题不大，批次大了导入频率就降低，而且导入性能也会更好。注意自己的任务规格去调优参数，避免采用过大的参数值导致OOM

#5

5分钟搞定MySQL/PostgreSQL/Oracle到StarRocks数据迁移同步-CloudCanal实战 | Clougence问答社区AskCUG
我按这篇文章改了参数，还改了，fe.conf配置（不过我不知道有没生效，没地方看），还是同样问题。

#6

值改的多大？任务规格有4G的话，可以改4K试试，另外有个停顿时间也可以控制。还有问题的话可以加下我微caizi19900527 帮你再确认下

京ICP备2022026421号-1