SMT迁移相关问题

为了更快的定位您的问题,请提供以下信息,谢谢
【详述】通过SMT工具进行迁移,源端为同一个数据库,配置多个同步任务报错,经过尝试发现最多只能配置18个任务,超过就报错:

Caused by: java.sql.SQLTransientConnectionException: connection-pool-x.x.x.x:3306 - Connection is not available, request timed out after 30000ms.
        at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:59) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:888) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:883) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at io.debezium.jdbc.JdbcConnection.connect(JdbcConnection.java:411) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        at com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:68) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
        ... 19 more

【背景】基于SMT做同步操作报错,尝试设置参数:
‘debezium.connect.timeout.ms’ = ‘300000’,
‘connect.timeout’ = ‘300000’,
‘connection.pool.size’ = ‘300’,
均无效果
源端数据库访问正常,经查,可能是hikaripool导致,需要修改pool参数,但这样是不是需要修改SMT源码?
想确认问题原因,如果是上面pool参数设置,能否将参数开放,或者已开放如何设置?

如果需要增加 cdc 的source 配置参数的话,可以在smt的conf文件的 [table-rule.x] 部分中增加如下配置

flink.cdc.xxx = yyy

其中xxx替换为对应的flink source配置参数。

这边添加了如下几个参数:

add flink-cdc

flink.cdc.connect.timeout = 60s
flink.cdc.connect.max-reties = 5
flink.cdc.connection.pool.size = 200

但是启动依旧报错:
Caused by: java.sql.SQLTransientConnectionException: connection-pool-192.168.8.1:3306 - Connection is not available, request timed out after 30000ms.
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.shaded.com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.source.connection.JdbcConnectionFactory.connect(JdbcConnectionFactory.java:59) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:888) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at io.debezium.jdbc.JdbcConnection.connection(JdbcConnection.java:883) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at io.debezium.jdbc.JdbcConnection.connect(JdbcConnection.java:411) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.debezium.DebeziumUtils.openJdbcConnection(DebeziumUtils.java:68) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.discoveryCaptureTables(MySqlSnapshotSplitAssigner.java:179) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.source.assigners.MySqlSnapshotSplitAssigner.open(MySqlSnapshotSplitAssigner.java:169) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.source.assigners.MySqlHybridSplitAssigner.open(MySqlHybridSplitAssigner.java:93) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at com.ververica.cdc.connectors.mysql.source.enumerator.MySqlSourceEnumerator.start(MySqlSourceEnumerator.java:87) ~[flink-sql-connector-mysql-cdc-2.4.0.jar:2.4.0]
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$start$1(SourceCoordinator.java:209) ~[flink-dist-1.15.4.jar:1.15.4]
at org.apache.flink.runtime.source.coordinator.SourceCoordinator.lambda$runInEventLoop$9(SourceCoordinator.java:406) ~[flink-dist-1.15.4.jar:1.15.4]
… 8 more