为了更快的定位您的问题，请提供以下信息，谢谢
【详述】
建立routine load消费kakfka数据导入starrocks表(明细表和主键表都有), 任务状态中途paused，报错kafka consume failed: 9747e4164410dc71-ec54c3bd379161ba, msg: fetch failed due to requested offset not available on the broker: Broker: Offset out of range (broker 1)，通过show routine load progress字段中对应的消费位点和kafka分区中offset对比发现并没有超出offset，日志也看不到详细情况
【背景】
建立starrocks明细表:
CREATE TABLE ods_log_weblog_pageview (
dt date comment ‘’
,id string comment ‘’
,vid string comment ‘’
,sys_lang string comment ‘’
,app_version string comment ‘’
,enli_user_id string comment ‘’
,channel string comment ‘’
,page_page_id string comment ‘’
,bit_vid bigint comment ‘’
,hostname string comment ‘’
,event_type string comment ‘’
,is_wifi string comment ‘’
,pvid string comment ‘’
,link_url string comment ‘’
,scm string comment ‘’
,prepage_site_id string comment ‘’
,sub_channel string comment ‘’
,visit_time datetime comment ‘’
,idfa string comment ‘’
,spm_curr string comment ‘’
,ab_version string comment ‘’
,prepage_page_id string comment ‘’
,user_id string comment ‘’
,os_type string comment ‘’
,prepage_pvid string comment ‘’
,event_name string comment ‘’
,spm_ref string comment ‘’
,supplier_id string comment ‘’
,country_id string comment ‘’
,cate_disp_id string comment ‘’
,item_code bigint comment ‘’
,rfx_no string comment ‘’
,page_site_id string comment ‘’
,timezone string comment ‘’
,prepage_pos string comment ‘’
,title string comment ‘’
,cate2_pub_id string comment ‘’
,page_id string comment ‘’
,device_name string comment ‘’
,cate1_pub_id string comment ‘’
,browser string comment ‘’
,s_clk string comment ‘’
,f_value string comment ‘’
,lang string comment ‘’
,user_agent string comment ‘’
,search_keyword string comment ‘’
,enli_date_source string comment ‘’
,hours string comment ‘’
,amount string comment ‘’
,os_version string comment ‘’
,vnum string comment ‘’
,session_id string comment ‘’
,ref_url string comment ‘’
,link_title string comment ‘’
,log_time datetime comment ‘’
,link_type string comment ‘’
,site string comment ‘’
,carrier string comment ‘’
,current_url string comment ‘’
,prepage_module_id string comment ‘’
,imei string comment ‘’
,resource_id string comment ‘’
,dh_item_code string comment ‘’
,ip_addr string comment ‘’
,event_time datetime comment ‘’
) ENGINE=OLAP
DUPLICATE KEY(dt, id,vid)
COMMENT “日志浏览事件表”
PARTITION BY date_trunc(‘day’, dt)
DISTRIBUTED BY HASH(id)
PROPERTIES (
“replication_num” = “3”,
“in_memory” = “false”,
“enable_persistent_index” = “true”,
“replicated_storage” = “true”,
“compression” = “LZ4”,
“storage_medium” = “SSD”
);
建立routine load
CREATE ROUTINE LOAD test.load_ods_log_weblog_pageview ON ods_log_weblog_pageview
columns (
dt
,id
,vid
,bit_vid
,sys_lang
,app_version
,enli_user_id
,channel
,page_page_id
,hostname
,event_type
,is_wifi
,pvid
,link_url
,scm
,prepage_site_id
,sub_channel
,visit_time
,idfa
,spm_curr
,ab_version
,prepage_page_id
,user_id
,os_type
,prepage_pvid
,event_name
,spm_ref
,supplier_id
,country_id
,cate_disp_id
,item_code
,rfx_no
,page_site_id
,timezone
,prepage_pos
,title
,cate2_pub_id
,page_id
,device_name
,cate1_pub_id
,browser
,s_clk
,f_value
,lang
,user_agent
,search_keyword
,enli_date_source
,hours
,amount
,os_version
,vnum
,session_id
,ref_url
,link_title
,log_time
,link_type
,site
,carrier
,current_url
,prepage_module_id
,imei
,resource_id
,dh_item_code
,ip_addr
,event_time
)
PROPERTIES
(
“desired_concurrent_number” = “3”,
“max_filter_ratio” = “1”,
“max_batch_interval”=“10”,
“format” = “json”,
“jsonpaths” = “[”$.dt","$.id","$.vid","$.bit_vid","$.sys_lang","$.app_version","$.enli_user_id","$.channel","$.page_page_id","$.hostname","$.event_type","$.is_wifi","$.pvid","$.link_url","$.scm","$.prepage_site_id","$.sub_channel","$.visit_time","$.idfa","$.spm_curr","$.ab_version","$.prepage_page_id","$.user_id","$.os_type","$.prepage_pvid","$.event_name","$.spm_ref","$.supplier_id","$.country_id","$.cate_disp_id","$.item_code","$.rfx_no","$.page_site_id","$.timezone","$.prepage_pos","$.title","$.cate2_pub_id","$.page_id","$.device_name","$.cate1_pub_id","$.browser","$.s_clk","$.f_value","$.lang","$.user_agent","$.search_keyword","$.enli_date_source","$.hours","$.amount","$.os_version","$.vnum","$.session_id","$.ref_url","$.link_title","$.log_time","$.link_type","$.site","$.carrier","$.current_url","$.prepage_module_id","$.imei","$.resource_id","$.dh_item_code","$.ip_addr","$.event_time"]"
)
FROM KAFKA
(
“kafka_broker_list”= “xxxxxx”,
“kafka_topic” = “kf-mds-log-weblog-pageview”,
“property.group.id” = “starRocks-routineload-ods_log_weblog_pageview”,
“property.kafka_default_offsets” = “OFFSET_BEGINNING”
);
【业务影响】任务无法实时导入，中途会paused
【是否存算分离】存算一体
【StarRocks版本】 3.1.11-34f131b
【集群规模】例如：3 fe, 3 be
【机器信息】CPU虚拟核/内存/网卡，例如：一台be16C/256G/万兆,二台be 16C/128G/万兆
【联系方式】小杨 - 601867126@qq.com
【附件】
- fe.log/beINFO/相应截图
谢谢，希望大佬们帮忙定位下