3.2.8版本paimon查询报错org.apache.paimon.fs.UnsupportedSchemeException

yuanzhangjun · 2024年07月3日 06:58

为了更快的定位您的问题，请提供以下信息，谢谢
【详述】paimon catalog 无法使用
【背景】s3作为存储，创建paimon catalog查询保持
【业务影响】
【是否存算分离】
【StarRocks版本】例如：3.2.8
【集群规模】例如：3fe（1 follower+2observer）+5be（fe与be混部）
【机器信息】4C32G
【联系方式】
【附件】

org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation for scheme ‘s3’ in the classpath

LIANGCHAOHUA · 2024年07月4日 03:13

用的AWS S3吗？权限检查一下。

yuanzhangjun · 2024年07月4日 03:50

权限没有问题，我使用3.2.6版本是可以的查询的，可以查询append表，但是无法查询pk表，
3.2.8版本就报这个问题，使用的是instance模式，但是报错确还需要aksk

2024-07-04 03:46:25.903Z WARN (starrocks-mysql-nio-pool-23|355) [HadoopUtils.getHadoopConfiguration():125] Could not find Hadoop configuration via any of the supported methods
2024-07-04 03:46:27.868Z WARN (starrocks-mysql-nio-pool-23|355) [StmtExecutor.execute():723] execute Exception, sql show databases
java.io.UncheckedIOException: org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation for scheme ‘s3’ in the classpath. Hadoop FileSystem also cannot access this path ‘s3://pre-bigdata-emr/data/flink/paimon’.
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:92) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at com.starrocks.connector.paimon.PaimonConnector.getPaimonNativeCatalog(PaimonConnector.java:124) ~[starrocks-fe.jar:?]
at com.starrocks.connector.paimon.PaimonConnector.getMetadata(PaimonConnector.java:131) ~[starrocks-fe.jar:?]
at com.starrocks.connector.CatalogConnector.getMetadata(CatalogConnector.java:36) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr$QueryMetadatas.getConnectorMetadata(MetadataMgr.java:94) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr.getOptionalMetadata(MetadataMgr.java:169) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr.getOptionalMetadata(MetadataMgr.java:148) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr.listDbNames(MetadataMgr.java:188) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ShowExecutor.handleShowDb(ShowExecutor.java:894) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ShowExecutor.execute(ShowExecutor.java:293) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleShow(StmtExecutor.java:1551) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:667) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:413) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:608) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:915) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_191]
Caused by: org.apache.paimon.fs.UnsupportedSchemeException: Could not find a file io implementation for scheme ‘s3’ in the classpath. Hadoop FileSystem also cannot access this path ‘s3://pre-bigdata-emr/data/flink/paimon’.
at org.apache.paimon.fs.FileIO.get(FileIO.java:385) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:89) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
… 19 more
Suppressed: java.io.IOException: One or more required options are missing.

Missing required options are:

s3.access-key
s3.secret-key
at org.apache.paimon.fs.FileIO.get(FileIO.java:348) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:89) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at com.starrocks.connector.paimon.PaimonConnector.getPaimonNativeCatalog(PaimonConnector.java:124) ~[starrocks-fe.jar:?]
at com.starrocks.connector.paimon.PaimonConnector.getMetadata(PaimonConnector.java:131) ~[starrocks-fe.jar:?]
at com.starrocks.connector.CatalogConnector.getMetadata(CatalogConnector.java:36) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr$QueryMetadatas.getConnectorMetadata(MetadataMgr.java:94) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr.getOptionalMetadata(MetadataMgr.java:169) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr.getOptionalMetadata(MetadataMgr.java:148) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr.listDbNames(MetadataMgr.java:188) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ShowExecutor.handleShowDb(ShowExecutor.java:894) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ShowExecutor.execute(ShowExecutor.java:293) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleShow(StmtExecutor.java:1551) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:667) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:413) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:608) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:915) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_191]
Suppressed: org.apache.hadoop.fs.s3a.AWSBadRequestException: getFileStatus on s3://pre-bigdata-emr/data/flink/paimon: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: A7MDHYXBKBDFZZA5; S3 Extended Request ID: NXrKRD3aURqguJlFqiZdrEF8Rg2kfV8SiqLp4X3i37pWoUfwnGo/TdrqToJW0Jug2y/YfTq6lh7UFmSX8f4Qpg==; Proxy: null), S3 Extended Request ID: NXrKRD3aURqguJlFqiZdrEF8Rg2kfV8SiqLp4X3i37pWoUfwnGo/TdrqToJW0Jug2y/YfTq6lh7UFmSX8f4Qpg==:400 Bad Request: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: A7MDHYXBKBDFZZA5; S3 Extended Request ID: NXrKRD3aURqguJlFqiZdrEF8Rg2kfV8SiqLp4X3i37pWoUfwnGo/TdrqToJW0Jug2y/YfTq6lh7UFmSX8f4Qpg==; Proxy: null)
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:247) ~[hadoop-aws-3.3.6.jar:?]
at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:174) ~[hadoop-aws-3.3.6.jar:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3746) ~[hadoop-aws-3.3.6.jar:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3652) ~[hadoop-aws-3.3.6.jar:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$exists$34(S3AFileSystem.java:4636) ~[hadoop-aws-3.3.6.jar:?]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547) ~[hadoop-common-3.3.6.jar:?]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528) ~[hadoop-common-3.3.6.jar:?]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449) ~[hadoop-common-3.3.6.jar:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2480) ~[hadoop-aws-3.3.6.jar:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2499) ~[hadoop-aws-3.3.6.jar:?]
at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4634) ~[hadoop-aws-3.3.6.jar:?]
at org.apache.paimon.fs.hadoop.HadoopFileIO.exists(HadoopFileIO.java:110) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at org.apache.paimon.fs.FileIOUtils.checkAccess(FileIOUtils.java:37) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at org.apache.paimon.fs.FileIO.get(FileIO.java:369) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:89) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:66) ~[paimon-bundle-0.7.0-incubating.jar:0.7.0-incubating]
at com.starrocks.connector.paimon.PaimonConnector.getPaimonNativeCatalog(PaimonConnector.java:124) ~[starrocks-fe.jar:?]
at com.starrocks.connector.paimon.PaimonConnector.getMetadata(PaimonConnector.java:131) ~[starrocks-fe.jar:?]
at com.starrocks.connector.CatalogConnector.getMetadata(CatalogConnector.java:36) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr$QueryMetadatas.getConnectorMetadata(MetadataMgr.java:94) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr.getOptionalMetadata(MetadataMgr.java:169) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr.getOptionalMetadata(MetadataMgr.java:148) ~[starrocks-fe.jar:?]
at com.starrocks.server.MetadataMgr.listDbNames(MetadataMgr.java:188) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ShowExecutor.handleShowDb(ShowExecutor.java:894) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ShowExecutor.execute(ShowExecutor.java:293) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.handleShow(StmtExecutor.java:1551) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:667) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:413) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:608) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:915) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_191]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: A7MDHYXBKBDFZZA5; S3 Extended Request ID: NXrKRD3aURqguJlFqiZdrEF8Rg2kfV8SiqLp4X3i37pWoUfwnGo/TdrqToJW0Jug2y/YfTq6lh7UFmSX8f4Qpg==; Proxy: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879) ~[aws-java-sdk-bundle-1.12.367.jar:?]

yuanzhangjun · 2024年07月4日 03:54

目前这两个版本都有一些问题，可以加一下你微信吗？

LIANGCHAOHUA · 2024年07月4日 04:35

就是3.2.6不用加这两个，3.2.8就不行了，提示要加上AK/SK？

LIANGCHAOHUA · 2024年07月4日 04:36

您在哪个企业微信群，我加一下。

yuanzhangjun · 2024年07月4日 05:56

社区1群-峻，或者直接加我yuan429808794

yuanzhangjun · 2024年07月4日 05:58

是的。3.2.6可以，3.2.7/3.2.8都不行