be 节点报：create table failed. status: Invalid argument: starlet err Invalid sys.root configuration provided!

U_1669262737894_3657 · 2024年01月6日 04:18

为了更快的定位您的问题，请提供以下信息，谢谢
【详述】
FE Leader 内存使用过高,集群无任何任务。之前通过routine load导入大量数据并创建物化视图表，之后FE时时宕机，后给FE节点添加内存。JAVA内存从28G修改到60G

【是否存算分离】是
【StarRocks版本】3.2.1-rc
【集群规模】3be+3fe
【机器信息】be/fe 32C 64G

fe 日志报

  2024-01-06 10:35:04,410 WARN (thrift-server-pool-3|226) [LeaderImpl.finishTask():236] task type: CREATE, status_code: RUNTIME_ERROR, create tablet failed, backendId: 10225, signature: 7325469
  2024-01-06 10:35:04,410 WARN (statistics meta manager|37) [LocalMetastore.waitForFinished():2135] fail to create tablet: 10224: [create tablet failed]
  2024-01-06 10:35:04,410 WARN (thrift-server-pool-9|273) [LeaderImpl.finishTask():236] task type: CREATE, status_code: RUNTIME_ERROR, create tablet failed, backendId: 10226, signature: 7325466
  2024-01-06 10:35:04,410 WARN (thrift-server-pool-8|272) [LeaderImpl.finishTask():188] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.141.4.203, be_port:9060, http_port:8040), task_type:CREATE, signature:7325470, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[create tablet failed]), report_version:17041933110000)
  2024-01-06 10:35:04,410 WARN (thrift-server-pool-4|268) [LeaderImpl.finishTask():236] task type: CREATE, status_code: RUNTIME_ERROR, create tablet failed, backendId: 10224, signature: 7325468
  2024-01-06 10:35:04,410 WARN (statistics meta manager|37) [StatisticsMetaManager.createSampleStatisticsTable():161] Failed to create sample statistics, fail to create tablet: 10224: [create tablet failed]
  2024-01-06 10:35:04,410 WARN (thrift-server-pool-8|272) [LeaderImpl.finishTask():222] cannot find task. type: CREATE, backendId: 10225, signature: 7325470
  2024-01-06 10:35:04,410 WARN (statistics meta manager|37) [StatisticsMetaManager.refreshStatisticsTable():338] create statistics table table_statistic_v1 failed
  2024-01-06 10:35:04,411 WARN (thrift-server-pool-9|273) [LeaderImpl.finishTask():188] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.141.4.204, be_port:9060, http_port:8040), task_type:CREATE, signature:7325467, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[create tablet failed]), report_version:17042510120000)
  2024-01-06 10:35:04,411 WARN (thrift-server-pool-9|273) [LeaderImpl.finishTask():222] cannot find task. type: CREATE, backendId: 10226, signature: 7325467

BE日志报

W0106 10:17:15.141175 122376 agent_task.cpp:224] create table failed. status: Invalid argument: starlet err Invalid sys.root configuration provided!
/build/starrocks/be/src/storage/protobuf_file.cpp:115 value_or_err_L115
/build/starrocks/be/src/storage/lake/tablet_manager.cpp:194 file.save(*metadata), signature: 7324091
W0106 10:17:15.142313 122378 agent_task.cpp:224] create table failed. status: Invalid argument: starlet err Invalid sys.root configuration provided!
/build/starrocks/be/src/storage/protobuf_file.cpp:115 value_or_err_L115
/build/starrocks/be/src/storage/lake/tablet_manager.cpp:194 file.save(*metadata), signature: 7324092
W0106 10:17:25.237133 122378 agent_task.cpp:224] create table failed. status: Invalid argument: starlet err Invalid sys.root configuration provided!
/build/starrocks/be/src/storage/protobuf_file.cpp:115 value_or_err_L115
/build/starrocks/be/src/storage/lake/tablet_manager.cpp:194 file.save(*metadata), signature: 7324097
W0106 10:17:25.237248 122376 agent_task.cpp:224] create table failed. status: Invalid argument: starlet err Invalid sys.root configuration provided!
/build/starrocks/be/src/storage/protobuf_file.cpp:115 value_or_err_L115
/build/starrocks/be/src/storage/lake/tablet_manager.cpp:194 file.save(*metadata), signature: 7324100
W0106 10:17:25.237376 122377 agent_task.cpp:224] create table failed. status: Invalid argument: starlet err Invalid sys.root configuration provided!
/build/starrocks/be/src/storage/protobuf_file.cpp:115 value_or_err_L115

fe.log/beINFO/相应截图
慢查询：
- Profile信息
- 并行度：show variables like ‘%parallel_fragment_exec_instance_num%’;
- pipeline是否开启：show variables like ‘%pipeline%’;
- be节点cpu和内存使用率截图
查询报错：
- query_dump，怎么获取query_dump文件
be crash
- be.out
- coredump，如何获取coredump
外表查询报错
- be.out和fe.warn.log

yuchen1019 · 2024年01月8日 12:12

麻烦提供下建表语句然后建议您可以升级下3.2.*的最新版本 rc是不正式的版本可能存在某些已解决的漏洞问题

U_1669262737894_3657 · 2024年01月9日 02:06

建了三个物化视图，有嵌套

CREATE MATERIALIZED VIEW `risk_ip_black_white_summary` (
`ip_addr`, `none`, `spite`, `harass`, `duidu`, `fraud`, `arbitrage`, `black`, `white`, `suspicious`, `update_time`)
COMMENT "MATERIALIZED_VIEW"
DISTRIBUTED BY HASH(`ip_addr`) BUCKETS 6 
REFRESH ASYNC START("2023-12-06 10:00:00") EVERY(INTERVAL 3 SECOND)
PROPERTIES (
"replicated_storage" = "true",
"replication_num" = "3",
"datacache.enable" = "true",
"enable_async_write_back" = "false",
"storage_volume" = "def_volume",
"colocate_with" = "ip_group"
)
AS SELECT `risk_ip_black_white_3`.`ip_addr`, sum(`risk_ip_black_white_3`.`none_tag`) AS `none`, 
sum(`risk_ip_black_white_3`.`spite_tag`) AS `spite`, sum(`risk_ip_black_white_3`.`harass_tag`) AS `harass`, 
sum(`risk_ip_black_white_3`.`duidu_tag`) AS `duidu`, sum(`risk_ip_black_white_3`.`fraud_tag`) AS `fraud`, 
sum(`risk_ip_black_white_3`.`arbitrage_tag`) AS `arbitrage`, sum(`risk_ip_black_white_3`.`is_black`) AS `black`, 
sum(`risk_ip_black_white_3`.`is_white`) AS `white`, sum(`risk_ip_black_white_3`.`is_suspicious`) AS `suspicious`, 
max(`risk_ip_black_white_3`.`update_time`) AS `update_time`
FROM `betcenter_risk`.`risk_ip_black_white_3`
GROUP BY `risk_ip_black_white_3`.`ip_addr`;



CREATE MATERIALIZED VIEW `risk_ip_all_site_agg_summary` (`ip_addr`, `sites`, `users`, `fail_req_users`, `none`, `spite`, `harass`, `duidu`, `fraud`, `arbitrage`, `black`, `white`, `suspicious`, `update_time`)
DISTRIBUTED BY HASH(`ip_addr`)
REFRESH ASYNC START("2023-12-01 00:00:00") EVERY(INTERVAL 2 SECOND)
PROPERTIES (
"replicated_storage" = "true",
"replication_num" = "3",
"storage_medium" = "HDD"
)
AS SELECT `a`.`ip_addr`, `a`.`sites`, `a`.`users`, `a`.`fail_req_users`, ifnull(`b`.`none`, 0) AS `none`, ifnull(`b`.`spite`, 0) AS `spite`, ifnull(`b`.`harass`, 0) AS `harass`, ifnull(`b`.`duidu`, 0) AS `duidu`, ifnull(`b`.`fraud`, 0) AS `fraud`, ifnull(`b`.`arbitrage`, 0) AS `arbitrage`, ifnull(`b`.`black`, 0) AS `black`, ifnull(`b`.`white`, 0) AS `white`, ifnull(`b`.`suspicious`, 0) AS `suspicious`, if(`a`.`update_time` > (ifnull(`b`.`update_time`, '1970-01-01 00:00:00')), `a`.`update_time`, ifnull(`b`.`update_time`, '1970-01-01 00:00:00')) AS `update_time`
FROM (SELECT `risa`.`ip_addr`, sum(`risa`.`count`) AS `sites`, count(DISTINCT `risa`.`users`) AS `users`, count(DISTINCT `risa`.`fail_req_users`) AS `fail_req_users`, max(`risa`.`update_time`) AS `update_time`
FROM `star_betcenter`.`risk_ip_site_agg` AS `risa`
GROUP BY `risa`.`ip_addr`) a LEFT OUTER JOIN `star_betcenter`.`risk_ip_black_white_summary` AS `b` ON `a`.`ip_addr` = `b`.`ip_addr`;


CREATE MATERIALIZED VIEW `risk_ip_site_agg_summary` (`ip_addr`, `site_code`, `site_users`, `site_fail_req_users`, `sites`, `users`, `fail_req_users`, `none`, `spite`, `harass`, `duidu`, `fraud`, `arbitrage`, `black`, `white`, `suspicious`, `site_update_time`, `update_time`)
DISTRIBUTED BY HASH(`ip_addr`) BUCKETS 12 
REFRESH ASYNC START("2023-12-06 10:00:00") EVERY(INTERVAL 3 SECOND)
PROPERTIES (
"replicated_storage" = "true",
"replication_num" = "3",
"storage_medium" = "HDD"
)
AS SELECT `a`.`ip_addr`, `a`.`site_code`, `a`.`users` AS `site_users`, `a`.`fail_req_users` AS `site_fail_req_users`, `b`.`sites`, `b`.`users`, `b`.`fail_req_users`, `b`.`none`, `b`.`spite`, `b`.`harass`, `b`.`duidu`, `b`.`fraud`, `b`.`arbitrage`, `b`.`black`, `b`.`white`, `b`.`suspicious`, `a`.`update_time` AS `site_update_time`, `b`.`update_time`
FROM (SELECT `risk_ip_site_agg`.`ip_addr`, `risk_ip_site_agg`.`site_code`, count(DISTINCT `risk_ip_site_agg`.`users`) AS `users`, count(DISTINCT `risk_ip_site_agg`.`fail_req_users`) AS `fail_req_users`, max(`risk_ip_site_agg`.`update_time`) AS `update_time`
FROM `star_betcenter`.`risk_ip_site_agg`
GROUP BY `risk_ip_site_agg`.`ip_addr`, `risk_ip_site_agg`.`site_code`) a INNER JOIN `star_betcenter`.`risk_ip_all_site_agg_summary` AS `b` ON `a`.`ip_addr` = `b`.`ip_addr`;

U_1669262737894_3657 · 2024年01月9日 02:17

我又看了一下版本，用的应该是发布版本，不是RC版本
Server version: 5.1.0 3.2.1-79ee91d

U_1669452058044_6098 · 2024年02月22日 07:50

大佬这个问题有解决么，遇到了同样的问题