为了更快的定位您的问题,请提供以下信息,谢谢
【详述】spark 读取 sr 表
all_id_df = spark.read.format(“starrocks”)
.option(“starrocks.fe.http.url”, “starrock.internal..com:8030")
.option(“starrocks.fe.jdbc.url”, "jdbc:mysql://starrock.internal..com:9030”)
.option(“starrocks.table.identifier”, “es.yh_danmu_detail_daily”)
.option(“starrocks.fenodes”, “starrock.internal.enlightent.com:8030”)
.option(“starrocks.batch.size”, “40960”)
.option(“starrocks.request.tablet.size”, “1”)
.option(“starrocks.exec.mem.limit”, “101474836480”)
.option(“user”, “***").option(“password”, "”)
.load()
【背景】
【业务影响】
【是否存算分离】是
【StarRocks版本】3.2.4
【connector版本】spark-connector 1.2
【集群规模】1fe +2be(fe与be混部)
df.show() 没有问题,
在执行 count 函数的时候, 会报错。
spark 前端错误:
24/05/29 15:04:50 ERROR [Thread-6] ScalaStarrocksRowRDD: Connect to StarRocks http://starrock.internal.enlightent.com:8030/api/es/yh_danmu_detail_daily/_query_plan failed.
Traceback (most recent call last):
File “”, line 1, in
File “/usr/lib/spark-current/python/pyspark/sql/dataframe.py”, line 680, in count
return int(self._jdf.count())
File “/usr/lib/spark-current/python/lib/py4j-0.10.9.3-src.zip/py4j/java_gateway.py”, line 1322, in call
File “/usr/lib/spark-current/python/pyspark/sql/utils.py”, line 111, in deco
return f(*a, **kw)
File “/usr/lib/spark-current/python/lib/py4j-0.10.9.3-src.zip/py4j/protocol.py”, line 328, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o259.count.
: com.starrocks.connector.spark.exception.ConnectedFailedException: Connect to http://starrock.internal.enlightent.com:8030/api/es/yh_danmu_detail_daily/_query_planfailed, status code is 500.
fe 错误日志:
com.starrocks.sql.parser.ParsingException: Getting syntax error at line 1, column 59. Detail message: Unexpected input ‘index’, the most similar input is {a legal identifier}.
at com.starrocks.sql.parser.ErrorHandler.syntaxError(ErrorHandler.java:40) ~[starrocks-fe.jar:?]
at org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:41) ~[antlr4-runtime-4.9.2.jar:4.9.2]
at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:544) ~[antlr4-runtime-4.9.2.jar:4.9.2]
at com.starrocks.sql.parser.SqlParser$1.reportInputMismatch(SqlParser.java:229) ~[starrocks-fe.jar:?]
at org.antlr.v4.runtime.DefaultErrorStrategy.reportError(DefaultErrorStrategy.java:139) ~[antlr4-runtime-4.9.2.jar:4.9.2]
at com.starrocks.sql.parser.StarRocksParser.selectItem(StarRocksParser.java:35263) ~[starrocks-fe.jar:?]
at com.starrocks.sql.parser.StarRocksParser.querySpecification(StarRocksParser.java:34266) ~[starrocks-fe.jar:?]
at com.starrocks.sql.parser.StarRocksParser.queryPrimary(StarRocksParser.java:33766) ~[starrocks-fe.jar:?]
at com.starrocks.sql.parser.StarRocksParser.queryNoWith(StarRocksParser.java:33472) ~[starrocks-fe.jar:?]
at com.starrocks.sql.parser.StarRocksParser.queryRelation(StarRocksParser.java:33349) ~[starrocks-fe.jar:?]
at com.starrocks.sql.parser.StarRocksParser.queryStatement(StarRocksParser.java:33281) ~[starrocks-fe.jar:?]
at com.starrocks.sql.parser.StarRocksParser.statement(StarRocksParser.java:1497) ~[starrocks-fe.jar:?]
at com.starrocks.sql.parser.StarRocksParser.singleStatement(StarRocksParser.java:758) ~[starrocks-fe.jar:?]
at com.starrocks.sql.parser.StarRocksParser.sqlStatements(StarRocksParser.java:649) ~[starrocks-fe.jar:?]
at com.starrocks.sql.parser.SqlParser.parseWithStarRocksDialect(SqlParser.java:104) ~[starrocks-fe.jar:?]
at com.starrocks.sql.parser.SqlParser.parse(SqlParser.java:66) ~[starrocks-fe.jar:?]
at com.starrocks.http.rest.TableQueryPlanAction.handleQuery(TableQueryPlanAction.java:212) ~[starrocks-fe.jar:?]
at com.starrocks.http.rest.TableQueryPlanAction.executeWithoutPassword(TableQueryPlanAction.java:168) ~[starrocks-fe.jar:?]
at com.starrocks.http.rest.RestBaseAction.execute(RestBaseAction.java:109) ~[starrocks-fe.jar:?]
at com.starrocks.http.rest.RestBaseAction.handleRequest(RestBaseAction.java:74) ~[starrocks-fe.jar:?]
at com.starrocks.http.HttpServerHandler.channelRead(HttpServerHandler.java:102) ~[starrocks-fe.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-all-4.1.61.Final.jar:4.1.61.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) ~[netty-all-4.1.61.Final.jar:4.1.61.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[netty-all-4.1.61.Final.jar:4.1.61.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.87.Final.jar:4.1.87.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)