字符串主键模型count(distinct key)很慢

【详述】字符串主键模型count(distinct key)很慢
【背景】查询慢
【业务影响】
【StarRocks版本】例如:2.4
【集群规模】例如:3fe(1 follower+2observer)+4be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:16C/64G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息,请补充下您的联系方式,例如:社区群4-小李或者邮箱,谢谢
【附件】

  • fe.log/beINFO/相应截图
  • 慢查询:
    • Profile信息
    • 并行度:show variables like ‘%parallel_fragment_exec_instance_num%’;

– bt.ods_uniqueid definition

CREATE TABLE test (
value varchar(64) NOT NULL COMMENT “”,
label varchar(8) NOT NULL COMMENT “”,
uniqueId bigint(20) NOT NULL COMMENT “”,
action varchar(20) NULL COMMENT “”,
createdAt datetime NULL COMMENT “”
) ENGINE=OLAP
PRIMARY KEY(value, label)
COMMENT “OLAP”
DISTRIBUTED BY HASH(value, label) BUCKETS 64
PROPERTIES (
“replication_num” = “3”,
“bloom_filter_columns” = “uniqueId”,
“in_memory” = “false”,
“storage_format” = “DEFAULT”,
“enable_persistent_index” = “false”
);

13个G的表下面的查询需要4分钟

select
label, count(distinct value)
from test
group by label;

这里主要是group by加上就慢

请问是什么问题?

PLAN FRAGMENT 0
OUTPUT EXPRS:2: label | 6: count
PARTITION: UNPARTITIONED

RESULT SINK

4:EXCHANGE

PLAN FRAGMENT 1
OUTPUT EXPRS:
PARTITION: HASH_PARTITIONED: 2: label

STREAM DATA SINK
EXCHANGE ID: 04
UNPARTITIONED

3:AGGREGATE (merge finalize)
| output: multi_distinct_count(6: count)
| group by: 2: label
|
2:EXCHANGE

PLAN FRAGMENT 2
OUTPUT EXPRS:
PARTITION: RANDOM

STREAM DATA SINK
EXCHANGE ID: 02
HASH_PARTITIONED: 2: label

1:AGGREGATE (update serialize)
| STREAMING
| output: multi_distinct_count(1: value)
| group by: 2: label
|
0:OlapScanNode
TABLE: ods_uniqueid_test
PREAGGREGATION: ON
partitions=1/1
rollup: ods_uniqueid_test
tabletRatio=64/64
tabletList=15570116,15570120,15570124,15570128,15570132,15570136,15570140,15570144,15570148,15570152 …
cardinality=141169844
avgRowSize=28.077595
numNodes=0

麻烦发下profile,获取Profile,通过Profile分析查询瓶颈

profile.txt (43.1 KB)

profile.txt发了

可以尝试设置
set new_planner_agg_stage = 3;

set new_planner_agg_stage = 4;

set new_planner_agg_stage = 3; 查询用了36秒,set new_planner_agg_stage = 4; 查询用了33秒