routine load 实时任务碰到ddl语句任务挂掉,并且丢数据

【详述】问题详细描述
routine load 实时任务碰到ddl语句任务挂掉
ddl语句data是null值,routine任务就直接挂掉了,在重启后,发现这个时间段的数据有丢失

【背景】做过哪些操作?
【业务影响】
【StarRocks版本】2.0.7
【集群规模3fe+6be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡 48C/256G/万兆

routine load 建任务语句如下

CREATE ROUTINE LOAD ods.routine_ods_receipt_tb_pay_receipt_txn ON ods_receipt_tb_pay_receipt_txn_ri
COLUMNS (
sharding_time=date_format(create_time,’%Y-%m-%d %H:%i:%S’),
trade_master_no,order_id,merchant_id,create_time
)
PROPERTIES
(
“desired_concurrent_number”=“10”,
“max_batch_interval” = “60”,
“max_batch_rows” = “300000”,
“max_batch_size” = “209715200”,
“strict_mode” = “false”,
“format” = “json”,
“strip_outer_array”=“true”,
“json_root”="$.data"
)
FROM KAFKA
(
“kafka_broker_list”= “kafka:9092”,
“kafka_topic” = “doirs_topic_ods_receipt_tb_pay_receipt_txn”,
“property.group.id” = “doirs_group_ods_receipt_tb_pay_receipt_txn”,
“property.kafka_default_offsets” = “OFFSET_BEGINNING”
);

您好,您的源数据是json对象还是json数组,看报错信息里面有出现让您设置strip_outer_array=false。strip_outer_array这个参数作用是将json数组打平。

数据源是json对象,是canal读取binlog的数据写到kafka的。

{“data”:[{“channel_pay_commit_no”:“PPC188051165597105620706676562”,“biz_order_no”:“20220623075735PP1910188038082012-1”,“channel_pay_request_no”:“20220623075735PP1910188038082012-1”,“quote_channel_pay_commit_no”:null,“main_request_type”:“1”,“request_type”:“pay”,“payment_type”:“20”,“channel_code”:“I_CODA_T01”,“channel_method_code”:“I_CODA_T01_CARRIERBILLING201_TRUEMOVETH_TH”,“channel_entity”:“T01”,“amount”:“35.0”,“paid_amount”:“35.0”,“country”:“TH”,“currency”:“THB”,“request_body”:"{“appId”:“fin-channel-exchange”,“appVersion”:“1.0”,“merchantTxnId”:“20220623075735PP1910188038082012-1”,“merchantId”:“010113832542899”,“channelOrderNo”:“PPC188051165597105620706676562”,“payOrderNo”:“20220623075735PP1910188038082012-1”,“productCode”:“10101”,“channelCode”:“CODA101”,“country”:“TH”,“methodCode”:“A15”,“methodSubCode”:“TruemoveTH”,“currency”:“THB”,“amount”:35.00,“userId”:“20220623P01C59165597056660405146912”,“purchaseInfo”:“Uggame Pay”,“remark”:“Uggame Pay”,“payerExtraDetails”:{“payerRemoteIp”:“171.103.175.210”},“terminalType”:“PMA”,“osType”:“ANDROID”,“acceptHeader”:"/",“userAgentHeader”:“Mozilla/5.0 (Linux; Android 12; M2011K2G Build/SKQ1.211006.001; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/102.0.5005.125 Mobile Safari/537.36 UShareitPaySDK”,“browserLanguage”:“th-TH”,“payerRemoteIp”:“171.103.175.210”,“channelCertificationNo”:“PPC188051165597105620706676562”}",“third_org_order_no”:null,“fourth_org_order_no”:null,“event_type”:null,“expire_time”:null,“status”:“9”,“mapping_code”:null,“mapping_msg”:null,“response_code”:null,“response_msg”:null,“batch_no”:null,“complete_time”:null,“create_time”:“2022-06-23 07:57:36.209624”,“modified_time”:“2022-06-23 07:57:36.209624”,“add_field1”:null,“add_field2”:null,“add_field3”:null,“add_field4”:null,“add_field5”:null,“add_field6”:null,“add_field7”:null}],“database”:“fin_channel_exchange_00”,“es”:1655971056000,“id”:29460930,“isDdl”:false,“mysqlType”:{“channel_pay_commit_no”:“varchar(64)”,“biz_order_no”:“varchar(64)”,“channel_pay_request_no”:“varchar(64)”,“quote_channel_pay_commit_no”:“varchar(64)”,“main_request_type”:“char(1)”,“request_type”:“varchar(32)”,“payment_type”:“varchar(4)”,“channel_code”:“varchar(64)”,“channel_method_code”:“varchar(128)”,“channel_entity”:“varchar(32)”,“amount”:“decimal(20,4)”,“paid_amount”:“decimal(20,4)”,“country”:“char(2)”,“currency”:“char(3)”,“request_body”:“mediumtext”,“third_org_order_no”:“varchar(64)”,“fourth_org_order_no”:“varchar(64)”,“event_type”:“varchar(16)”,“expire_time”:“int(10)”,“status”:“tinyint(1)”,“mapping_code”:“varchar(64)”,“mapping_msg”:“varchar(512)”,“response_code”:“varchar(64)”,“response_msg”:“varchar(512)”,“batch_no”:“varchar(64)”,“complete_time”:“timestamp(6)”,“create_time”:“timestamp(6)”,“modified_time”:“timestamp(6)”,“add_field1”:“varchar(256)”,“add_field2”:“varchar(64)”,“add_field3”:“varchar(128)”,“add_field4”:“varchar(32)”,“add_field5”:“varchar(64)”,“add_field6”:“varchar(512)”,“add_field7”:“varchar(128)”},“old”:null,“pkNames”:[“channel_pay_commit_no”,“request_type”],“sql”:"",“sqlType”:{“channel_pay_commit_no”:12,“biz_order_no”:12,“channel_pay_request_no”:12,“quote_channel_pay_commit_no”:12,“main_request_type”:1,“request_type”:12,“payment_type”:12,“channel_code”:12,“channel_method_code”:12,“channel_entity”:12,“amount”:3,“paid_amount”:3,“country”:1,“currency”:1,“request_body”:2005,“third_org_order_no”:12,“fourth_org_order_no”:12,“event_type”:12,“expire_time”:4,“status”:-6,“mapping_code”:12,“mapping_msg”:12,“response_code”:12,“response_msg”:12,“batch_no”:12,“complete_time”:93,“create_time”:93,“modified_time”:93,“add_field1”:12,“add_field2”:12,“add_field3”:12,“add_field4”:12,“add_field5”:12,“add_field6”:12,“add_field7”:12},“table”:“tb_funding_channel_commit_order_22”,“ts”:1655971056286,“type”:“INSERT”}

实际数据是这样的格式,我们读取data下的数据,data是一个数组,但是如果是ddl语句, data就是null,就会报错

您好,我们当前不支持ddl语句的。具体可以看下这里https://docs.starrocks.com/zh-cn/main/loading/PrimaryKeyLoad#%E7%A4%BA%E4%BE%8B-2-1

我们不是要支持ddl,我们是接收到ddl,任务就直接挂掉了,并且数据有丢失。 这个有版本说让任务正常么,现在是没办法用,mysql 加个字段,加个索引,任务就挂了,数据就少了