数据从hive 走brokeload到sr,但是有的能成功有的会出现失败
问题:测试的hive数据来源有两个,分别是hbase 同步到hive,另一个是从其他hive表同步到hive ,hive的写入方式都是使用hivewirter。在此处用hive1表示数据来源为hbase,hive2为数据源为其他hive表,其中hive1和hive2使用一样的建表语句,格式为parquet,除了表名以外,其余都一样,并且hive1和hive2 和starrocks的表字段一样,数据也一样,不同点是hive1和hive2都多一个rowkey字段。导入的brokeload脚本也是一样的
执行部分:
hive1-》brokeload to sr 出现报错,报错信息如下
hive2-》brokeload to sr 提示成功
brokeload 脚本为
LOAD LABEL btest_load_2023818
(
DATA INFILE("hdfs://sfbd/user/hive/dm/dm_test/dm_test_dtl/")
INTO TABLE broker_test
FORMAT AS “parquet”
(mobile,rangetype,grade_val,consignee_risk_customer,consignor_risk_customer,is_hige_sensitive_customer,consignee_risk_customer_top,consignor_risk_customer_top,ishigh,consignedtm,silence_rank,consignor_box_rate,order_cancell_rate,last_consigned_date,last_consigned_city,consignor_self_rate,consignor_changtime,consignor_total_fee,consignor_limit_top3,consignor_weight_top3,is_delete)
SET (mobile=mobile,rangetype=rangetype,grade_val=grade_val,consignee_risk_customer=split(regexp_replace(consignee_risk_customer,’[\[|\]]’, ‘’),’,’),consignor_risk_customer=split(regexp_replace(consignor_risk_customer,’[\[|\]]’, ‘’),’,’),is_hige_sensitive_customer=is_hige_sensitive_customer,consignee_risk_customer_top=consignee_risk_customer_top,consignor_risk_customer_top=consignor_risk_customer_top,ishigh=ishigh,consignedtm=consignedtm,silence_rank=silence_rank,consignor_box_rate=consignor_box_rate,order_cancell_rate=order_cancell_rate,last_consigned_date=last_consigned_date,last_consigned_city=last_consigned_city,consignor_self_rate=consignor_self_rate,consignor_changtime=consignor_changtime,consignor_total_fee=consignor_total_fee,consignor_limit_top3=split(regexp_replace(consignor_limit_top3,’[\[|\]]’, ‘’),’,’),consignor_weight_top3=split(regexp_replace(consignor_weight_top3,’[\[|\]]*’, ‘’),’,’)is_delete=is_delete)where is_delete=‘0’ and mobile is not null)
WITH BROKER hdfs_broker PROPERTIES(“max_filter_ratio” = “0.9”,“timeout” = “259200”);