【详述】建立routine load消费kafka数据导入starrocks表(明细表和主键表都有), 有一个topic对应的routine load 任务任务状态一直running(其他routine load任务正常),但是kafka中的数据却一条都进不了starrocks中,通过show routine load 查看Statistic: {“receivedBytes”:0,“errorRows”:0,“committedTaskNum”:0,“loadedRows”:0,“loadRowsRate”:0,“abortedTaskNum”:0,“totalRows”:0,“unselectedRows”:0,“receivedBytesRate”:0,“taskExecuteTimeMs”:1}一直都是这样没有变化,创建routine load时kafka_offsets设置OFFSET_BEGINNING或者OFFSET_END,LatestSourcePosition都是一样的也不会变化,另外能确认kafka对应的3分区是实时有数据进kafka的,日志也看不到详细情况。
【背景】
*建立starrocks主键表:(为避免字段影响,只选取其中几个字段建表)
use test;
CREATE TABLE ods_sales_promo
(
promo_id
BIGINT NOT NULL COMMENT “活动id”,
promo_type_id
BIGINT NULL COMMENT “促销类型ID 0-折扣、1-直降”,
promo_display_element_id
BIGINT NULL COMMENT “活动展示元素ID”,
table_name
varchar(65533) NULL COMMENT “源表名”
) ENGINE=OLAP
PRIMARY KEY(promo_id
)
DISTRIBUTED BY HASH(promo_id
)
PROPERTIES (
“replication_num” = “3”,
“in_memory” = “false”,
“enable_persistent_index” = “true”,
“replicated_storage” = “true”,
“compression” = “LZ4”
);
建立routine load:
CREATE ROUTINE LOAD test.load_ods_sales_promo ON ods_sales_promo
columns (
promo_id
,promo_type_id
,promo_display_element_id
,table_name
),
where table_name = ‘td_promo’
PROPERTIES
(
“desired_concurrent_number” = “3”,
“max_batch_interval”=“10”,
“format” = “json”,
“strict_mode” = “false”,
“jsonpaths” = “[”$.promo_id","$.promo_type_id","$.promo_display_element_id","$.table_name"]"
)
FROM KAFKA
(
“kafka_broker_list”= “xxx:9092”,
“kafka_topic” = “db-mysql-cms-etl”,
“property.group.id” = “starRocks-routineload-ods_sales_promo”,
“kafka_partitions” = “3”,
“kafka_offsets” = “OFFSET_END”
– “property.kafka_default_offsets” = “OFFSET_END” --确定此表数据只在3分区
);
【业务影响】任务无法实时导入,一条数据都没有进starrocks
【是否存算分离】存算一体
【StarRocks版本】 3.1.11-34f131b
【集群规模】例如:3 fe, 3 be
【机器信息】CPU虚拟核/内存/网卡,例如:一台be16C/256G/万兆,二台be 16C/128G/万兆
【联系方式】社区群19-乐云山
【附件】
![image](https://cdn-forum.starrocks.com/original/2X/0/0a5b356a5666b1cb93ae4e0d958c8ac9c17e2f67.png)
be.INFO (56.7 MB)
fe-10.log (61.6 MB)