SMT迁移相关问题

💬 StarRocks 用户问答 常见 FAQ
#1

为了更快的定位您的问题，请提供以下信息，谢谢
【详述】通过SMT工具进行迁移，源端为同一个数据库，配置多个同步任务报错，经过尝试发现最多只能配置18个任务，超过就报错：

Caused by: java.sql.SQLTransientConnectionException: connection-pool-x.x.x.x:3306 - Connection is not available, request timed out after 30000ms.
        at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:59) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:888) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:883) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at io.debezium.jdbc.JdbcConnection.connect(JdbcConnection.java:411) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:68) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        ... 19 more

【背景】基于SMT做同步操作报错，尝试设置参数：
‘debezium.connect.timeout.ms’ = ‘300000’,
‘connect.timeout’ = ‘300000’,
‘connection.pool.size’ = ‘300’,
均无效果
源端数据库访问正常，经查，可能是hikaripool导致，需要修改pool参数，但这样是不是需要修改SMT源码？
想确认问题原因，如果是上面pool参数设置，能否将参数开放，或者已开放如何设置？

#2

如果需要增加 cdc 的source 配置参数的话，可以在smt的conf文件的 [table-rule.x] 部分中增加如下配置

flink.cdc.xxx = yyy

其中xxx替换为对应的flink source配置参数。

#3

这边添加了如下几个参数：

add flink-cdc

flink.cdc.connect.timeout = 60s
flink.cdc.connect.max-reties = 5
flink.cdc.connection.pool.size = 200

但是启动依旧报错：
Caused by: java.sql.SQLTransientConnectionException: connection-pool-192.168.8.1:3306 - Connection is not available, request timed out after 30000ms.
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:59) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:888) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:883) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at io.debezium.jdbc.JdbcConnection.connect(JdbcConnection.java:411) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:68) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.discoveryCaptureTables(MySqlSnapshotSplitAssigner.java:179) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.open(MySqlSnapshotSplitAssigner.java:169) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.source.assigners.MySqlHybridSplitAssigner.open(MySqlHybridSplitAssigner.java:93) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator.start(MySqlSourceEnumerator.java:87) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$start$1(SourceCoordinator.java:209) ~[flink-dist-1.15.4.jar:1.15.4]
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$runInEventLoop$9(SourceCoordinator.java:406) ~[flink-dist-1.15.4.jar:1.15.4]
… 8 more

#4

这个问题解决了么，遇到同样问题，求解，只能并行18个

#5

换个同步平台，dinky，streampark

1赞

京ICP备2022026421号-1