【详述】flink load 在 exactly-once模式无法写入到starrocks并且任务直接卡住,且在fe和be没有相关报错日志
【背景】写入模式切换成at-least-once后写入正常
【业务影响】无法写入数据
【StarRocks版本】2.5.2
【集群规模】例如：3fe（1 follower+2observer）+3be（fe与be混部）
【机器信息】CPU虚拟核/内存/网卡，例如：48C/64G/万兆
【表模型】明细模型
【导入或者导出方式】flink connector版本1.2.7,flink版本1.13.5
【联系方式】344970961@qq.com
【附件】建表语句 CREATE TABLE test.user_log_detail (
createTime DATETIME NULL COMMENT “”,
userId BIGINT NULL COMMENT “”,
userType STRING NULL COMMENT “”,
pageId STRING NULL COMMENT “”,
tableId STRING NULL COMMENT “”,
account STRING NULL COMMENT “”,
appName SMALLINT NULL COMMENT “”,
appType STRING NULL COMMENT “”,
appStyle STRING NULL COMMENT “”,
appVersion STRING NULL COMMENT “”,
busiId STRING NULL COMMENT “”,
canFinger STRING NULL COMMENT “”,
city STRING NULL COMMENT “”,
cpus STRING NULL COMMENT “”,
deviceId STRING NULL COMMENT “”,
district STRING NULL COMMENT “”,
event SMALLINT NULL COMMENT “”,
eventDuration BIGINT NULL COMMENT “”,
gsm STRING NULL COMMENT “”,
iSP STRING NULL COMMENT “”,
isVPN STRING NULL COMMENT “”,
longLat STRING NULL COMMENT “”,
messageId STRING NULL COMMENT “”,
mobileType STRING NULL COMMENT “”,
nation STRING NULL COMMENT “”,
netConType STRING NULL COMMENT “”,
province STRING NULL COMMENT “”,
recvTime DATETIME NULL COMMENT “”,
requestParam STRING NULL COMMENT “”,
resolution STRING NULL COMMENT “”,
sensorList STRING NULL COMMENT “”,
sessionId STRING NULL COMMENT “”,
setupTime STRING NULL COMMENT “”,
simStatus STRING NULL COMMENT “”,
source SMALLINT NULL COMMENT “”,
systemVersion STRING NULL COMMENT “”,
timeZone STRING NULL COMMENT “”,
transTime DATETIME NULL COMMENT “”,
loadDate DATE NULL COMMENT “”
) DUPLICATE KEY(createTime,userId,userType,pageId,tableId)
DISTRIBUTED BY HASH(userId) BUCKETS 3 ;
sink表配置
CREATE TABLE starrocks_user_log_detail (
account STRING,
appName SMALLINT,
appType STRING,
appStyle STRING,
appVersion STRING,
busiId STRING,
canFinger STRING,
city STRING,
cpus STRING,
createTime STRING,
deviceId STRING,
district STRING,
event SMALLINT,
eventDuration BIGINT,
gsm STRING,
iSP STRING,
isVPN STRING,
longLat STRING,
messageId STRING,
mobileType STRING,
nation STRING,
netConType STRING,
pageId STRING,
province STRING,
recvTime STRING,
requestParam STRING,
resolution STRING,
sensorList STRING,
sessionId STRING,
setupTime STRING,
simStatus STRING,
source SMALLINT,
systemVersion STRING,
tableId STRING,
timeZone STRING,
transTime STRING,
userId BIGINT,
userType STRING,
loadDate STRING
) WITH (
‘connector’ = ‘starrocks’,
‘jdbc-url’ = ‘jdbc:mysql://ip:9030/test?characterEncoding=utf-8&serverTimezone=Asia/Shanghai’,
‘load-url’ = ‘ip:8030’,
‘database-name’ = ‘test’,
‘table-name’ = ‘user_log_detail’,
‘username’ = ‘xx’,
‘password’ = ‘xx’,
‘sink.properties.format’ = ‘json’,
‘sink.properties.strip_outer_array’ = ‘true’,
‘sink.max-retries’ = ‘1’,
‘sink.semantic’ = ‘at-least-once’,
‘sink.properties.timeout’ = ‘600’,
‘sink.version’ = ‘v2’,
‘sink.buffer-flush.max-bytes’ = ‘67108864’,
‘sink.buffer-flush.max-rows’ = ‘64000’,
‘sink.buffer-flush.interval-ms’ = ‘10000’
)
flink 上没任何输出么？
没有 直接卡住了
可以在 flink 日志中查看下导入状态 去日志里面搜下 看有类似的日志打印吗 ？http://$fe:${http_port}/api/$db/$tbl/_stream_load ，有的话表示成功触发了 Stream Load 任务了 （flink底层走的stream load），任务结果也会打印在 flink 日志中的 可以看下详情
没有打印stream load任务信息，应该是直接卡在执行stream load任务之前了