paimon catalog 配置obs warehouse路径查询报错

💬 StarRocks 用户问答
#1

为了更快的定位您的问题，请提供以下信息，谢谢
【详述】

  1. 创建paimon hive catalog
    CREATE EXTERNAL CATALOG paimon_catalog
    COMMENT ‘访问paimon hive’
    PROPERTIES
    (
    “type” = “paimon”,
    “paimon.catalog.type”=“hive”,
    “paimon.catalog.warehouse”=“obs://xxx/user/hive/warehouse”,
    “hive.metastore.uris” = “thrift://xxx:9083”,
    “aws.s3.enable_ssl” = “false”,
    “aws.s3.enable_path_style_access” = “true”,
    “aws.s3.endpoint” = “https://.myhuaweicloud.com”,
    “aws.s3.access_key” = “<iam_user_access_key>”,
    “aws.s3.secret_key” = “<iam_user_secret_key>”
    );
  2. 查询已存在的paimon表报错
    ERROR 1064 (HY000): org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation for scheme ‘obs’ in the classpath. Hadoop FileSystem also cannot access this path ‘obs://xxx/user/hive/warehouse’.

【是否存算分离】存算分离 obs
【StarRocks版本】3.3.0
【集群规模】单节点测试
【联系方式】 847692946@qq.com

#2

通过实践发现可行方式-参考obs文档对接hadoop配置：
a. be/fe的conf下默认有core-site.xml ，添加配置


fs.s3.impl
org.apache.hadoop.fs.s3a.S3AFileSystem
默认配置


fs.obs.impl
org.apache.hadoop.fs.obs.OBSFileSystem


fs.AbstractFileSystem.obs.impl
org.apache.hadoop.fs.obs.OBS


fs.obs.access.key
xxxx
HuaweiCloud Access Key Id


fs.obs.secret.key
xxxx
HuaweiCloud Secret Access Key


fs.obs.endpoint
obs.ap-southeast-3.myhuaweicloud.com
HuaweiCloud Endpoint


b. 拷贝obs jar hadoop-huaweicloud-3.1.1-hw-46.1.jar 到be/fe相关lib下
cp /opt/hadoopclient/HDFS/hadoop/share/hadoop/tools/lib/hadoop-huaweicloud-3.1.1-hw-46.1.jar /opt/flink-lakehouse/olap_starrocks/StarRocks-3.3.0/fe/lib/
cp /opt/hadoopclient/HDFS/hadoop/share/hadoop/tools/lib/hadoop-huaweicloud-3.1.1-hw-46.1.jar /opt/flink-lakehouse/olap_starrocks/StarRocks-3.3.0/be/lib/paimon-reader-lib/

总结疑问：
配置paimon catalog前实践了配置hive catalog
CREATE EXTERNAL CATALOG hive_catalog
COMMENT ‘访问hive’
PROPERTIES
(
“type” = “hive”,
“hive.metastore.type” = “hive”,
– 高可用地址用逗号分隔
“hive.metastore.uris” = “thrift://192.168.89.21:9083,thrift://192.168.89.15:9083”,
“aws.s3.enable_ssl” = “false”,
“aws.s3.enable_path_style_access” = “true”,
“aws.s3.endpoint” = “https://obs.ap-southeast-3.myhuaweicloud.com”,
“aws.s3.access_key” = “<iam_user_access_key>”,
“aws.s3.secret_key” = “<iam_user_secret_key>”
);
在hive catalog中查询并没有异常，顺利查到了位于obs的表数据，但是paimon catalog 异常。 观察区别，paimon catalog中需要指定“paimon.catalog.warehouse” 。 是否是这里的实现存在问题？

京ICP备2022026421号-1