事实表使用主键模型按天分区,并开启主键持久化。每天一千万数据,主键内存占用较大。
数据写入时间都是当前时间,如果可以按小时分区,内存中只需要缓存近一两个小时的主键。
但是跨天之后,想把昨天的小时分区变成天分区。
目前不支持这种方法是设置分区。您这个是什么版本,开启持久化,每天千万数据,主键内存占用是多大,方便的话请发下建表语句。
一天快要结束时,主键内存占用800M 。
使用版本为 2.5.12 。
建表语句如下:
create table dwd_event (
`dt` DateTime ,
`uid` int ,
pk_enum TINYINT ,
pk_hash BIGINT ,
field String ,
field String ,
field bigint ,
field bigint ,
field DateTime ,
field DateTime ,
field bigint ,
field int ,
field string ,
field string ,
field string ,
field string ,
field string ,
field string ,
field string ,
field string ,
field string ,
field string ,
field string ,
field string ,
field tinyint ,
field string ,
field string ,
field string ,
field string ,
field string ,
field int ,
field string ,
field string ,
field tinyint ,
field tinyint ,
field tinyint ,
field string ,
field string ,
field string ,
field string ,
field string ,
field string ,
field string ,
field float ,
field string ,
field float ,
field float ,
field tinyint ,
field String ,
field DateTime ,
field float ,
field tinyint ,
field string ,
field float ,
field string ,
field INT ,
field float ,
field float ,
field string ,
field float ,
field float ,
field float ,
field float ,
field tinyint ,
field tinyint ,
field string ,
field string ,
field float ,
field int ,
field string ,
field string ,
field tinyint ,
field float ,
field float ,
field string ,
field string ,
field DateTime ,
field string ,
field string ,
field string ,
field string ,
field string ,
field int ,
field int ,
field int ,
field string ,
field bigint ,
field DateTime ,
field int ,
field string ,
field string ,
field string ,
field int ,
field int ,
field int ,
field string ,
field DATE,
field STRING,
field STRING,
field datetime NULL
) ENGINE=olap
PRIMARY KEY (dt, uid, pk_enum, pk_hash)
PARTITION BY RANGE (dt) (
START ("2019-12-01") END ("2024-01-01") EVERY (INTERVAL 1 day)
)
DISTRIBUTED BY HASH(`uid`) BUCKETS 6
PROPERTIES (
"colocate_with" = "cg",
"replication_num" = "2",
"dynamic_partition.enable" = "true",
"dynamic_partition.time_unit" = "DAY",
"dynamic_partition.end" = "3",
"dynamic_partition.prefix" = "p",
"dynamic_partition.buckets" = "6",
"enable_persistent_index" = "true"
);