使用 StarRocks 查询 hive catalog 中存储于 hdfs 上的外表

[U_1666161925568_2650]

1、环境介绍

当前实时任务接收所有前端上报的数据，中间通过flink任务将数据流拆解为实时任务和离线任
务，近三天的数据实时写⼊CK，⽽离线数据为flink接收数据流之后⼏乎不做处理通过⽂件流的形式写⼊
华为云的hive中，因此CK⾥的数据和hive中的数据⼀致，除了华为云之后，现在还有线下的规模较⼩的
CDH集群和规模⽐较⼤的集群(简称⼤集群)。

CDH集群上主要部署flink实时任务，同时为了分解华为云的压⼒，会把部分数据从线上的华为云同步到线下的⼤集群上。

2、 业务介绍

⽬前线下CK中近3天的宽表数据，主要做⼀些实时的维度的PV和UV的计算(只查询近3天的数据)，表
关联的场景⽐如漏⽃、留存等场景以及有3天前的数据是查询线上的华为云hive数据。

3、 starrocks⽅案

短期计划

短期计划是打算在本地再搭⼀套starrocks，利⽤starrocks通过hive catalog查询表数据的功能，快速实现报表查询功能，由于通过hive catalog查询华为云会产⽣⼤量的⽹络开销，对⽬前带宽的是很⼤的考验，因此⽬前计划查询的是⼤集群上的数据，

⻓期计划

⽤starrocks取代hive和CK，实现真正的流批⼀体

4、查询目标

1. 实时任务写⼊性能测试
2. 数据查询性能测试

由于hive的查询性能和starrocks差距太⼤，就不做对⽐分析了

5、硬件规格

6、参数设置

7、数据规模

表结构：按照日期分区，按照产品ID排序

单月数据量：

new_user：6990135条
daily_active_user：92155306

event_1:10354689125
event_2:777605922
event_3:14453971633

单年数据量（13个月）

new_user：61970661条
daily_active_user：1066125517条

event_1:95232274288
event_2:5070695497
event_3:117199721038

8、测试 SQL-流量趋势附件2-1：流量趋势 一年 三并发测试-按维度 ip=21.log (1.3 MB)

见附件 Profile

9、测试结果

⼀年的数据量同时只跑⼀个的单发任务

⼀年的数据量同时跑三个的任务

10、其它

1）pipeline

2）explain_costs

explain_costs.log (376.4 KB)

[yuchen1019]

大数据集指的是本地的cdh？ 当前是有什么问题呢 查的慢还是？ 我看cache也开了 优化配置都是蛮ok的

[andy_xu_007]

大集群是hadoop开源搭的集群，规模挺大的，当前的问题如下：
1，查询的速度不是很理想，想知道还有哪些优化的空间
2，查询会失败，我们希望哪怕查的慢一些，也不能接收失败
我们希望的效果如下：
一年的数据量，并行三个，在争取在两分钟之内结束，并且没有报错
一个月的数据量，并行是个，正确在10s之内结束，并且没有报错
我们想知道还有哪些优化的空间，比如外表转内表，资源隔离等等，期待您的回复

[U_1666161925568_2650]

抱歉，之前贴执行时间截图的时候出错，特此勘误：

9、测试结果

⼀年的数据量同时只跑⼀个的单发任务

⼀年的数据量同时跑三个的任务【三台节点均会报内存溢出】

[U_1695089838477_1400]

使用cache还需要设置 SET global enable_populate_block_cache = true;

[U_1695089838477_1400]

set global enable_spill = true 可以开启spill，应该能解决并发下的oom问题

[U_1695089838477_1400]

可以用物化视图提升查询性能，

以hive_catalog_big_cluster.new_user表为例

	SELECT  'e582aa8c-0617-4a8b-8d96-2975a2c7566d' AS taskid
	       ,'_td_current_country'                  AS dim_name
	       ,geo_info.ip_country                    AS dim_value
	       ,'new' kpi_name
	       ,COUNT(DISTINCT tdid)                   AS kpi_value
	FROM hive_catalog_big_cluster.new_user NEW
	WHERE partitionday >= '20220403'
	AND partitionday <= '20230430'
	AND appkey = 'A9A91F57C3914B0D9F7F0AF5EE5E8ED4'
	GROUP BY  geo_info.ip_country

可以基于如下查询构建MV

	SELECT  'e582aa8c-0617-4a8b-8d96-2975a2c7566d' AS taskid
	       ,'_td_current_country'                  AS dim_name
	       ,geo_info.ip_country                    AS dim_value
	       ,'new' kpi_name
	       ,bitmap_union(to_bitmap(tdid))          AS kpi_value
	FROM hive_catalog_big_cluster.new_user NEW
	GROUP BY  geo_info.ip_country, partitionday, appeky

关于mv的具体内容可以参考文档 https://docs.starrocks.io/en-us/latest/using_starrocks/Materialized_view

[U_1666161925568_2650]

增加了配置，三并发查询一年的数据还是会报错，辛苦再给看下呢

SET global enable_populate_block_cache = true;
set global enable_spill = true;

[yuchen1019]

机器的具体配置麻烦再给出一下呢

[U_1666161925568_2650]

1、Configure Info

sys_log_roll_interval=DAY
tablet_sched_consecutive_full_clone_delay_sec=180
enable_sql_blacklist=false
routine_load_task_timeout_second=60
max_allowed_in_element_num_of_delete=10000
label_keep_max_num=1000
consistency_tablet_meta_check_interval_ms=7200000
max_stream_load_timeout_second=259200
quorum_publish_wait_time_ms=5000
metadata_journal_skip_bad_journal_ids=
export_checker_interval_second=5
azure_blob_path=
yarn_client_path=/data/hadoop/StarRocks-3.1.2/fe/lib/yarn-client/hadoop/bin/yarn
storage_usage_soft_limit_percent=90
spark_load_submit_timeout_second=300
enable_refresh_hive_partitions_statistics=true
http_max_chunk_size=8192
default_replication_num=3
enable_iceberg_custom_worker_thread=false
check_java_version=true
statistic_cache_thread_pool_size=10
storage_usage_soft_limit_reserve_bytes=214748364800
iceberg_metadata_disk_cache_capacity=2147483648
dump_log_modules=[query]
small_file_dir=/data/hadoop/StarRocks-3.1.2/fe/small_files
binlog_ttl_second=1800
azure_blob_shared_key=
aws_s3_path=
azure_blob_endpoint=
enable_safe_mode=false
query_port=9030
routine_load_task_consume_second=15
authentication_ldap_simple_user_search_attr=uid
enable_hms_events_incremental_sync=false
prepared_transaction_default_timeout_second=86400
max_distribution_pruner_recursion_depth=100
tablet_sched_slot_num_per_path=8
disable_hadoop_load=false
pipe_scheduler_interval_millis=1000
enable_metric_calculator=true
http_max_header_size=32768
sys_log_dir=/data/hadoop/StarRocks-3.1.2/fe/log
authentication_ldap_simple_server_port=389
enable_auto_tablet_distribution=true
pipe_listener_interval_millis=1000
histogram_max_sample_row_count=10000000
tablet_create_timeout_second=10
export_max_bytes_per_be_per_task=268435456
profile_info_format=default
big_query_log_roll_interval=DAY
tablet_sched_max_not_being_scheduled_interval_ms=900000
enable_hms_parallel_process_evens=true
publish_version_interval_ms=10
edit_log_port=9010
enable_routine_load_lag_metrics=false
semi_sync_collect_statistic_await_seconds=30
tablet_checker_partition_batch_num=500
authentication_ldap_simple_bind_root_pwd=
iceberg_metadata_cache_disk_path=/data/hadoop/StarRocks-3.1.2/fe/caches/iceberg
thrift_backlog_num=1024
aws_s3_region=
enable_show_external_catalog_privilege=true
bdbje_replica_ack_timeout_second=10
binlog_max_size=9223372036854775807
ignore_meta_check=false
hdfs_file_system_expire_seconds=300
period_of_auto_resume_min=5
http_backlog_num=1024
iceberg_metadata_memory_cache_capacity=536870912
lake_autovacuum_stale_partition_threshold=12
locale=zh_CN.UTF-8
audit_log_roll_num=90
cloud_native_hdfs_url=
thrift_rpc_retry_times=3
qe_max_connection=1024
backup_plugin_path=/tools/trans_file_tool/trans_files.sh
edit_log_roll_num=50000
remote_file_metadata_load_concurrency=32
load_parallel_instance_num=15
task_runs_max_history_number=10000
enable_collect_full_statistic=true
max_agent_tasks_send_per_be=10000
tablet_sched_always_force_decommission_replica=false
enable_background_refresh_resource_table_metadata=false
load_straggler_wait_second=300
desired_max_waiting_jobs=1024
task_ttl_second=86400
consistency_check_start_time=23
enable_validate_password=false
master_sync_policy=SYNC
http_web_page_display_hardware=true
alter_max_worker_threads=4
thrift_server_max_worker_threads=4096
bdbje_log_level=INFO
agent_task_resend_wait_time_ms=5000
lake_compaction_simple_selector_threshold_versions=10
iceberg_table_refresh_expire_sec=86400
plugin_enable=true
alter_scheduler_interval_millisecond=10000
authorization_enable_priv_collection_cache=true
max_download_task_per_be=0
replica_sync_policy=SYNC
routine_load_kafka_timeout_second=12
db_used_data_quota_update_interval_secs=300
big_query_log_dir=/data/hadoop/StarRocks-3.1.2/fe/log
max_load_timeout_second=259200
big_query_log_delete_age=7d
yarn_config_dir=/data/hadoop/StarRocks-3.1.2/fe/lib/yarn-config
bdbje_replay_cost_percent=150
hive_max_split_size=67108864
max_query_retry_time=2
statistic_collect_concurrency=3
statistic_auto_collect_large_table_interval=43200
aws_s3_iam_role_arn=
dump_log_delete_age=7d
label_keep_max_second=259200
thrift_client_timeout_ms=5000
metadata_journal_max_batch_size_mb=10
enable_pipeline_load=true
brpc_connection_pool_size=16
min_load_timeout_second=1
max_routine_load_batch_size=4294967296
meta_delay_toleration_second=300
lake_compaction_history_size=12
mysql_nio_backlog_num=1024
authorization_enable_admin_user_protection=false
tablet_delete_timeout_second=2
check_consistency_default_timeout_second=600
log_roll_size_mb=1024
iceberg_table_refresh_threads=128
enable_create_partial_partition_in_batch=false
max_small_file_size_bytes=1048576
enable_check_db_state=true
tablet_sched_disable_colocate_balance=false
hive_meta_load_concurrency=4
capacity_used_percent_high_water=0.75
authentication_ldap_simple_bind_base_dn=
plugin_dir=/data/hadoop/StarRocks-3.1.2/fe/plugins
disable_load_job=false
txn_rollback_limit=100
tablet_sched_balance_load_disk_safe_threshold=0.5
max_planner_scalar_rewrite_num=100000
broker_load_default_timeout_second=14400
backup_job_default_timeout_ms=86400000
tablet_sched_min_clone_task_timeout_sec=180
remote_file_cache_ttl_s=129600
iceberg_metadata_cache_max_entry_size=8388608
storage_usage_hard_limit_reserve_bytes=107374182400
hive_meta_cache_refresh_interval_s=7200
enable_auth_check=true
statistic_analyze_status_keep_second=259200
statistic_auto_collect_small_table_size=5368709120
lake_compaction_fail_history_size=12
metadata_failure_recovery=false
thrift_rpc_timeout_ms=10000
lake_autovacuum_max_previous_versions=0
statistic_sample_collect_rows=200000
dynamic_partition_enable=true
ssl_truststore_password=
alter_max_worker_queue_size=4096
statistic_manager_sleep_time_sec=60
enable_dict_optimize_routine_load=false
load_checker_interval_second=5
azure_blob_sas_token=
privilege_max_role_depth=16
hive_meta_cache_ttl_s=86400
ignore_materialized_view_error=false
enable_replicated_storage_as_default_engine=true
report_queue_size=100
max_tolerable_backend_down_num=0
enable_statistic_collect=true
enable_dict_optimize_stream_load=true
ssl_keystore_location=
statistic_collect_too_many_version_sleep=600000
min_routine_load_lag_for_metrics=10000
max_partitions_in_one_batch=4096
max_create_table_timeout_second=600
statistic_auto_analyze_start_time=00:00:00
tmp_dir=/data/hadoop/StarRocks-3.1.2/fe/temp_dir
spark_home_default_dir=/data/hadoop/StarRocks-3.1.2/fe/lib/spark2x
dump_log_dir=/data/hadoop/StarRocks-3.1.2/fe/log
external_table_commit_timeout_ms=10000
export_task_pool_size=5
enable_execute_script_on_frontend=true
export_task_default_timeout_second=7200
sys_log_level=INFO
mysql_service_io_threads_num=4
empty_load_as_error=true
thrift_rpc_strict_mode=true
iceberg_worker_num_threads=64
lake_compaction_max_tasks=-1
enable_display_shadow_partitions=false
routine_load_pulsar_timeout_second=12
statistic_collect_max_row_count_per_query=5000000000
max_running_rollup_job_num_per_table=1
max_routine_load_task_num_per_be=16
statistic_auto_collect_small_table_interval=0
tablet_sched_disable_balance=false
thrift_server_queue_size=4096
spark_load_default_timeout_second=86400
iceberg_metadata_disk_cache_expiration_seconds=604800
aws_s3_endpoint=
big_query_log_modules=[query]
spark_launcher_log_dir=/data/hadoop/StarRocks-3.1.2/fe/log/spark_launcher_log
tablet_sched_disable_colocate_overall_balance=true
enable_udf=false
statistic_auto_collect_sample_threshold=0.3
max_connection_scheduler_threads_num=4096
enable_collect_query_detail_info=false
bdbje_reserved_disk_size=536870912
statistic_update_interval_sec=86400
lake_compaction_score_selector_min_score=10.0
auto_increment_cache_size=100000
sys_log_delete_age=7d
http_max_initial_line_length=4096
stream_load_default_timeout_second=600
qe_slow_log_ms=5000
cloud_native_storage_type=S3
enable_experimental_temporary_table=false
star_mgr_meta_sync_interval_sec=600
authentication_chain=[native]
max_hive_partitions_per_rpc=5000
heartbeat_mgr_blocking_queue_size=1024
histogram_mcv_size=100
enable_shuffle_load=true
enable_vectorized_file_load=true
audit_log_delete_age=30d
dump_log_roll_num=10
tablet_sched_balance_load_score_threshold=0.1
enable_starrocks_external_table_auth_check=true
max_per_node_grep_log_limit=500000
max_agent_task_threads_num=4096
max_upload_task_per_be=0
max_automatic_partition_number=4096
run_mode=shared_nothing
meta_dir=/data/hadoop/StarRocks-3.1.2/fe/meta
lake_compaction_selector=ScoreSelector
edit_log_type=BDB
history_job_keep_max_second=604800
hms_events_batch_size_per_rpc=500
enable_materialized_view=true
enable_iceberg_metadata_disk_cache=false
es_state_sync_interval_second=10
hive_meta_store_timeout_s=10
heartbeat_mgr_threads_num=8
stream_load_profile_collect_second=10
max_small_file_number=100
tablet_sched_storage_cooldown_second=-1
mysql_server_version=5.1.0
enable_new_publish_mechanism=false
enable_experimental_mv=true
statistic_max_full_collect_data_size=107374182400
frontend_address=0.0.0.0
enable_remote_script=false
ignore_invalid_privilege_authentications=false
enable_password_reuse=true
storage_usage_hard_limit_percent=95
audit_log_modules=[slow_query, query]
iceberg_metadata_memory_cache_expiration_seconds=86500
spark_resource_path=
background_refresh_metadata_interval_millis=600000
recover_with_empty_tablet=false
background_refresh_metadata_time_secs_since_last_access_secs=86400
max_broker_load_job_concurrency=15
enable_statistic_collect_on_first_load=true
lake_autovacuum_partition_naptime_seconds=180
metadata_journal_queue_size=1000
dynamic_partition_check_interval_seconds=600
aws_s3_access_key=
hdfs_read_buffer_size_kb=8192
cluster_name=StarRocks Cluster
lake_compaction_simple_selector_threshold_seconds=300
sys_log_verbose_modules=[]
background_refresh_file_metadata_concurrency=4
enable_sync_publish=false
task_runs_concurrency=4
shard_group_clean_threshold_sec=3600
authentication_kerberos_service_principal=
min_bytes_per_broker_scanner=67108864
tablet_sched_max_clone_task_timeout_sec=7200
max_routine_load_task_concurrent_num=5
statistic_collect_interval_sec=300
task_check_interval_second=14400
max_stream_load_batch_size_mb=100
max_backend_down_time_second=3600
heartbeat_retry_times=3
statistic_full_collect_buffer=20971520
enable_materialized_view_spill=true
load_profile_info_reserved_num=500
eliminate_shuffle_load_by_replicated_storage=true
max_mysql_service_task_threads_num=4096
insert_load_default_timeout_second=3600
enable_local_replica_selection=false
enable_token_check=true
enable_strict_storage_medium_check=false
max_dynamic_partition_num=500
ignore_unknown_log_id=false
hms_process_events_parallel_num=4
heartbeat_timeout_second=5
enable_schedule_insert_query_by_row_count=true
label_clean_interval_second=14400
tablet_sched_checker_interval_seconds=20
access_control=native
export_running_job_num_limit=15
max_running_txn_num_per_db=100
audit_log_roll_interval=DAY
aws_s3_use_instance_profile=false
spark_dpp_version=1.0.0
brpc_idle_wait_max_time=10000
safe_mode_checker_interval_sec=5
privilege_max_total_roles_per_user=64
transaction_clean_interval_second=30
consistency_check_end_time=4
thrift_rpc_max_body_size=-1
mysql_service_nio_enabled=true
statistic_auto_analyze_end_time=23:59:59
slow_lock_threshold_ms=3000
sys_log_roll_mode=SIZE-MB-1024
http_port=8030
max_bdbje_clock_delta_ms=5000
alter_table_timeout_second=86400
hive_meta_cache_refresh_min_threads=50
remote_file_cache_refresh_interval_s=60
tablet_sched_colocate_balance_high_prio_backends=[]
dump_log_roll_interval=DAY
tablet_stat_update_interval_second=300
tablet_sched_max_scheduling_tablets=10000
enable_authentication_kerberos=false
tablet_sched_max_migration_task_sent_once=1000
profile_info_reserved_num=500
tablet_sched_repair_delay_factor_second=60
hdfs_write_buffer_size_kb=1024
big_query_log_roll_num=10
statistic_cache_columns=100000
task_runs_ttl_second=86400
tablet_sched_be_down_tolerate_time_s=900
aws_s3_use_aws_sdk_default_behavior=false
tablet_sched_max_balancing_tablets=500
hms_events_polling_interval_ms=5000
statistic_collect_query_timeout=3600
ssl_keystore_password=
sys_log_to_console=false
sys_log_roll_num=10
drop_backend_after_decommission=true
statistic_check_expire_partition=true
aws_s3_secret_key=
histogram_buckets_size=64
lake_autovacuum_parallel_partitions=8
authentication_ldap_simple_server_host=
ssl_key_password=
catalog_trash_expire_second=86400
broker_client_timeout_ms=120000
statistic_dict_columns=100000
routine_load_scheduler_interval_millisecond=10000
authentication_kerberos_service_key_tab=
slow_lock_log_every_ms=3000
start_with_incomplete_meta=false
bdbje_lock_timeout_second=1
rpc_port=9020
task_runs_queue_length=500
catalog_try_lock_timeout_ms=5000
ssl_truststore_location=
create_table_max_serial_replicas=128
enable_load_volume_from_conf=true
metadata_journal_max_batch_cnt=100
log_plan_cancelled_by_crash_be=true
cluster_id=-1
enable_decimal_v3=true
expr_children_limit=10000
lake_compaction_sorter=ScoreSorter
cloud_native_meta_port=6090
jaeger_grpc_endpoint=
lake_autovacuum_grace_period_minutes=5
priority_networks=
enable_background_refresh_connector_metadata=true
audit_log_dir=/data/hadoop/StarRocks-3.1.2/fe/log
bdbje_heartbeat_timeout_second=30
aws_s3_external_id=
authentication_ldap_simple_bind_root_dn=
statistic_auto_collect_ratio=0.8
histogram_sample_ratio=0.1
vectorized_load_enable=true
tablet_sched_colocate_be_down_tolerate_time_s=43200
replica_ack_policy=SIMPLE_MAJORITY
auth_token=
bdbje_cleaner_threads=1
lake_compaction_simple_selector_min_versions=3

2、Variable Info

SQL_AUTO_IS_NULL=false
activate_all_roles_on_login=true
allow_default_partition=false
analyze_mv=sample
auto_increment_increment=1
autocommit=true
big_query_log_cpu_second_threshold=480
big_query_log_scan_bytes_threshold=10737418240
big_query_log_scan_rows_threshold=1000000000
broadcast_right_table_scale_factor=10.0
broadcast_row_limit=15000000
cbo_cte_max_limit=10
cbo_cte_reuse=true
cbo_cte_reuse_rate=1.15
cbo_debug_alive_backend_number=0
cbo_enable_dp_join_reorder=true
cbo_enable_greedy_join_reorder=true
cbo_enable_low_cardinality_optimize=true
cbo_max_reorder_node=50
cbo_max_reorder_node_use_dp=10
cbo_max_reorder_node_use_exhaustive=4
cbo_prune_shuffle_column_rate=0.1
cbo_prune_subfield=true
cbo_push_down_aggregate=global
cbo_push_down_aggregate_mode=-1
cbo_push_down_distinct_below_window=true
cbo_reorder_threshold_use_exhaustive=6
cbo_use_correlated_join_estimate=true
cbo_use_lock_db=false
cbo_use_nth_exec_plan=0
character_set_client=utf8
character_set_connection=utf8
character_set_database=utf8
character_set_results=utf8
character_set_server=utf8
chunk_size=4096
collation_connection=utf8_general_ci
collation_database=utf8_general_ci
collation_server=utf8_general_ci
connector_io_tasks_per_scan_operator=16
connector_io_tasks_slow_io_latency_ms=50
connector_scan_use_query_mem_ratio=0.3
consistent_hash_virtual_number=32
count_distinct_column_buckets=1024
default_rowset_type=alpha
default_table_compression=lz4_frame
disable_colocate_join=false
disable_function_fold_constants=false
disable_join_reorder=false
div_precision_increment=4
enable_adaptive_sink_dop=true
enable_big_query_log=true
enable_cbo_table_prune=false
enable_collect_table_level_scan_stats=true
enable_column_expr_predicate=true
enable_connector_adaptive_io_tasks=true
enable_count_star_optimization=true
enable_deliver_batch_fragments=true
enable_distinct_column_bucketization=false
enable_exchange_pass_through=true
enable_exchange_perf=false
enable_filter_unused_columns_in_scan_stage=true
enable_global_runtime_filter=true
enable_groupby_use_output_alias=false
enable_hive_column_stats=true
enable_hive_metadata_cache_with_insert=false
enable_incremental_mv=false
enable_insert_strict=true
enable_load_profile=false
enable_local_shuffle_agg=true
enable_materialized_view_rewrite=true
enable_materialized_view_single_table_view_delta_rewrite=false
enable_materialized_view_union_rewrite=true
enable_materialized_view_view_delta_rewrite=true
enable_multicolumn_global_runtime_filter=false
enable_mv_optimizer_trace_log=false
enable_mv_planner=false
enable_optimizer_trace_log=false
enable_outer_join_reorder=true
enable_parallel_merge=true
enable_pipeline_engine=true
enable_pipeline_query_statistic=true
enable_plan_validation=true
enable_populate_block_cache=true
enable_predicate_reorder=false
enable_profile=true
enable_prune_complex_types=true
enable_query_cache=false
enable_query_debug_trace=false
enable_query_dump=false
enable_query_queue_load=false
enable_query_queue_select=false
enable_query_queue_statistic=false
enable_rbo_table_prune=false
enable_rewrite_groupingsets_to_union_all=false
enable_rewrite_simple_agg_to_meta_scan=false
enable_rewrite_sum_by_associative_rule=true
enable_rule_based_materialized_view_rewrite=true
enable_runtime_adaptive_dop=false
enable_scan_block_cache=true
enable_shared_scan=false
enable_show_all_variables=false
enable_simplify_case_when=true
enable_sort_aggregate=false
enable_spill=true
enable_sql_digest=false
enable_strict_type=false
enable_sync_materialized_view_rewrite=true
enable_tablet_internal_parallel=true
event_scheduler=OFF
exec_mem_limit=2147483648
force_schedule_local=false
forward_to_leader=false
full_sort_late_materialization=false
full_sort_max_buffered_bytes=16777216
full_sort_max_buffered_rows=1024000
global_runtime_filter_build_max_size=67108864
global_runtime_filter_probe_min_selectivity=0.5
global_runtime_filter_probe_min_size=102400
group_concat_max_len=65535
hash_join_push_down_right_table=true
hdfs_backend_selector_hash_algorithm=consistent
hdfs_backend_selector_scan_range_shuffle=false
hive_partition_stats_sample_size=3000
hudi_mor_force_jni_reader=false
init_connect=
innodb_read_only=true
interactive_timeout=3600
interpolate_passthrough=true
io_tasks_per_scan_operator=4
join_implementation_mode_v2=auto
language=/starrocks/share/english/
license=Apache License 2.0
load_mem_limit=0
load_transmission_compression_type=NO_COMPRESSION
log_rejected_record_num=0
lower_case_table_names=0
materialized_view_rewrite_mode=DEFAULT
max_allowed_packet=33554432
max_parallel_scan_instance_num=-1
max_pipeline_dop=64
max_pushdown_conditions_per_column=-1
max_scan_key_num=-1
nested_mv_rewrite_max_level=3
net_buffer_length=16384
net_read_timeout=60
net_write_timeout=60
new_planner_agg_stage=0
new_planner_optimize_timeout=3000
parallel_exchange_instance_num=-1
parallel_fragment_exec_instance_num=1
parse_tokens_limit=3500000
partial_update_mode=auto
performance_schema=false
pipeline_dop=0
pipeline_profile_level=1
pipeline_sink_dop=0
prefer_compute_node=false
prefer_cte_rewrite=false
profile_limit_fold=true
profile_timeout=2
query_cache_agg_cardinality_limit=5000000
query_cache_entry_max_bytes=4194304
query_cache_entry_max_rows=409600
query_cache_force_populate=false
query_cache_hot_partition_num=3
query_cache_size=1048576
query_cache_type=0
query_delivery_timeout=300
query_excluding_mv_names=
query_including_mv_names=
query_mem_limit=0
query_queue_concurrency_limit=1
query_queue_cpu_used_permille_limit=900
query_queue_fresh_resource_usage_interval_ms=5000
query_queue_max_queued_queries=12
query_queue_mem_used_pct_limit=0.7
query_queue_pending_timeout_second=600
query_timeout=30000
range_pruner_max_predicate=100
resource_group=
resource_group_id=0
rpc_http_min_size=2147482624
runtime_adaptive_dop_max_block_rows_per_driver_seq=16384
runtime_adaptive_dop_max_output_amplification_factor=0
runtime_filter_early_return_selectivity=0.05
runtime_filter_on_exchange_node=false
runtime_filter_scan_wait_time=20
runtime_join_filter_push_down_limit=1024000
runtime_profile_report_interval=10
scan_or_to_union_limit=4
scan_or_to_union_threshold=50000000
scan_use_query_mem_ratio=0.3
select_ratio_threshold=0.15
single_node_exec_plan=false
spill_encode_level=7
spill_mem_limit_threshold=0.8
spill_mem_table_num=2
spill_mem_table_size=104857600
spill_mode=auto
spill_operator_max_bytes=1048576000
spill_operator_min_bytes=10485760
spillable_operator_mask=-1
sql_dialect=StarRocks
sql_mode=ONLY_FULL_GROUP_BY
sql_quote_show_create=true
sql_safe_updates=0
sql_select_limit=9223372036854775807
statistic_collect_parallel=1
storage_engine=olap
streaming_preaggregation_mode=auto
system_time_zone=Asia/Shanghai
tablet_internal_parallel_mode=auto
time_zone=Asia/Shanghai
transaction_isolation=REPEATABLE-READ
transmission_compression_type=NO_COMPRESSION
transmission_encode_level=7
tx_isolation=REPEATABLE-READ
tx_visible_wait_timeout=10
use_compute_nodes=-1
use_page_cache=true
version=5.1.0
version_comment=3.1.2-4f3a2ee
wait_timeout=28800
window_partition_mode=1

[U_1666161925568_2650]

目前的tdid类型是varchar

bitmap_union(to_bitmap(tdid)) 语法不支持，只能使用全局字典吗？