为了更快的定位您的问题,请提供以下信息,谢谢
【详述】基于hive catalog外部表创建分区异步物化视图,查询外部表是不走物化视图,仍然查询外部表
【背景】升级前版本为3.2.3,查询hive外部表能查询改写命中物化视图,升级到V3.2.11后,所有外部表查询都不能命中物化视图。是不是V3.2.11有严重的bug?
【业务影响】所有生产环境的物化视图都无法命中,导致查询效率低
【是否存算分离】否
【StarRocks版本】3.2.11
【集群规模】3fe(2 follower+1 leader)+3be(fe与be分开部署)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】社区群10-Touché
【附件】
1、已开启物化视图查询改写:enable_materialized_view_rewrite: true
2、物化视图创建语句:
use test;
CREATE MATERIALIZED VIEW mv_async_fix7dt_a_fk_gj_rent_rate_dtl_tbl_test
PARTITION BY (dt
)
DISTRIBUTED BY HASH(cust_id
)
REFRESH ASYNC START(“2024-07-20 10:00:00”) EVERY(INTERVAL 1 DAY)
PROPERTIES (
“replicated_storage” = “true”,
“replication_num” = “1”,
“storage_medium” = “HDD”
)
AS SELECT * FROM test
.adl
.a_fk_gj_rent_rate_dtl_tbl_test
WHERE dt
>= cast(date_format(date_add(current_date(), interval -1 year),’%Y%m%d’) as int)
;
基表为hive catalog外部表:test
.adl
.a_fk_gj_rent_rate_dtl_tbl_test
,表分区字段类型为bigint,字段格式参考示例:20240802。
3、外部表查询
explain
select * from test
.adl
.a_fk_gj_rent_rate_dtl_tbl_test
where dt=20240823;
查询的分区为物化视图中的分区,但执行计划中显示查的是基表,而非物化视图
0:HdfsScanNode
TABLE: a_fk_gj_rent_rate_dtl_tbl_test
PARTITION PREDICATES: 42: dt = 20240823
partitions=1/6
cardinality=1
avgRowSize=42.0
limit: 200
4、按照官网排查物化视图无法命中的原因
TRACE LOGS MV
select *
from test
.adl
.a_fk_gj_rent_rate_dtl_tbl_test
where dt=20240823;
部分内容如下:显示有物化视图,但是查询改写失败
1ms| [MV TRACE] [PREPARE GLOBAL] Table/MaterializedView a_fk_gj_rent_rate_dtl_tbl_test has related materialized views: [MvId{dbId=332201, id=829035}]
1ms| [MV TRACE] [PREPARE GLOBAL] Choose 1/1 mvs after user config
1ms| [MV TRACE] [PREPARE GLOBAL] Choose 1/1 valid mvs after checking valid
1ms| [MV TRACE] [REWRITE mv_async_fix7dt_a_fk_gj_rent_rate_dtl_tbl_test] invalid query plan
1ms| [MV TRACE] [PREPARE GLOBAL] There are no valid related mvs for the query plan
2ms| [MV TRACE] [REWRITE TF_MV_TEXT_MATCH_REWRITE_RULE] [InMemo:false] TEXT_BASED_REWRITE matched mvs: []
2ms| [MV TRACE] [REWRITE TF_MV_TEXT_MATCH_REWRITE_RULE] [InMemo:false] OptToAstMap is empty, no try to rewrite sub-query again
4ms| Query cannot be rewritten, please check the trace logs to find more information.
Tracer Cost: 128us
升级前版本为 3.2.3,就能命中物化视图,物化视图创建语句没区别
已在社区微信群反馈,目前未收到反馈
经反复验证发现是时间函数导致的,估计是V3.2.11版本有bug,不能像低版本V3.2.3那样正确解析,导致物化视图查询改写失败。当前还没发现解决办法,各位大佬们有碰到类似问题吗,怎么解决的呢?
– WHERE dt
>= cast(date_format(‘2024-02-20 09:35:22’, ‘%Y%m%d’) as int) – cast和data_format函数能命中物化视图
– WHERE dt
>= cast(date_format(date_add(‘2025-02-20’, interval -1 year),’%Y%m%d’) as int) – date_add能命中物化视图
– WHERE dt
>= cast(date_format(date_add(now(), interval -1 year),’%Y%m%d’) as int) – now()不能命中物化视图
– WHERE dt
>= cast(date_format(date_add(current_timestamp(), interval -1 year),’%Y%m%d’) as int) – CURDATE()不能命中物化视图
– WHERE dt
>= cast(date_format(date_add(current_date(), interval -1 year),’%Y%m%d’) as int) – current_date()不能命中物化视图
– WHERE dt
>= cast(date_format(date_add(CURDATE(), interval -1 year),’%Y%m%d’) as int) – CURDATE()不能命中物化视图
– WHERE dt
>= cast(date_format(date_add(DATE(current_timestamp()), interval -1 year),’%Y%m%d’) as int) – 不能命中物化视图
– WHERE dt
>= cast(date_format(date_add(DATE(UTC_TIMESTAMP()), in