为了更快的定位您的问题，请提供以下信息，谢谢
【详述】检查集群统计信息发现任务状态是Pending，查看meta信息，大部分表都没有正常更新
【背景】升级到2.5.22版本两周后发现这个问题
【业务影响】集群SQL查询性能普遍下降，影响所有业务
【是否存算分离】否
【StarRocks版本】2.5.22
【集群规模】3fe（1 follower+2observer）+10be（独立部署）
【机器信息】CPU虚拟核/内存/网卡， 32C/128G/万兆
【联系方式】社区群12-金谡 jinsu@moojing.com
【附件】
- fe.log/beINFO/相应截图
任务状态查询
mysql> SHOW ANALYZE JOB;
±------±---------±------±--------±-------±---------±-----------±--------±-------------±-------+
| Id | Database | Table | Columns | Type | Schedule | Properties | Status | LastWorkTime | Reason |
±------±---------±------±--------±-------±---------±-----------±--------±-------------±-------+
| 10146 | ALL | ALL | ALL | SAMPLE | SCHEDULE | {} | PENDING | None | |
±------±---------±------±--------±-------±---------±-----------±--------±-------------±-------+
1 row in set (0.00 sec)
mysql> admin show frontend config like ‘enable_collect_full_statistic’
-> ;
±------------------------------±-----------±------±--------±----------±--------+
| Key | AliasNames | Value | Type | IsMutable | Comment |
±------------------------------±-----------±------±--------±----------±--------+
| enable_collect_full_statistic | [] | true | boolean | true | |
±------------------------------±-----------±------±--------±----------±--------+
1 row in set (0.01 sec)
meta信息查询
mysql> SHOW STATS META;
±-------------------------±--------------------------------------------------------±--------±-------±--------------------±-----------±--------+
| Database | Table | Columns | Type | UpdateTime | Properties | Healthy |
±-------------------------±--------------------------------------------------------±--------±-------±--------------------±-----------±--------+
| eanalyze | item_patch_wy_v3_1727058577 | ALL | FULL | 2024-09-23 10:33:47 | {} | 100% |
| qa_db | qa_itemattr2_custom | ALL | FULL | 2024-10-06 23:07:21 | {} | 100% |
| qa_db | backup_20241007_replace_datas_in202309_itemattr2 | ALL | FULL | 2024-10-07 10:02:24 | {} | 100% |
| eanalyze | itemattr2 | ALL | FULL | 2024-08-12 01:55:48 | {} | 92% |
| eanalyze | item_patch_bejf_1725987803 | ALL | FULL | 2024-09-11 01:14:31 | {} | 100% |
| eanalyze | item_patch_xx_v3_1728632515 | ALL | FULL | 2024-10-11 15:45:09 | {} | 100% |
| eanalyze | item_patch_kskj_v3_1725589291 | ALL | FULL | 2024-09-14 10:37:51 | {} | 100% |
| eanalyze | item2_view_110119 | ALL | FULL | 2024-08-12 00:34:19 | {} | 100% |
| eanalyze | item_patch_mrk_1725796552 | ALL | FULL | 2024-09-08 19:58:49 | {} | 100% |
| eanalyze | brands_gome | ALL | FULL | 2024-08-10 23:38:24 | {} | 100% |
| eanalyze | item_patch_moodytiger_1725612835 | ALL | FULL | 2024-09-06 16:57:55 | {} | 100% |
| qa_db | qa_20230531_itemattr2_custom | ALL | FULL | 2024-09-24 15:20:13 | {} | 100% |
| eanalyze | item_patch_ystywyjs_v3_1728440040 | ALL | FULL | 2024-10-09 10:18:14 | {} | 100% |
| eanalyze | item_patch_dytm_1699945448 | ALL | FULL | 2024-08-12 01:25:22 | {} | 100% |
| eanalyze | item_patch_sxinner_v3_1728372569 | ALL | FULL | 2024-10-08 15:31:53 | {} | 100% |
| eanalyze | item_patch_1728711875 | ALL | FULL | 2024-10-12 13:45:00 | {} | 100% |
检查fe日志，发现有任务错误信息：
Oct 11, 2024 5:02:55 PM com.github.benmanes.caffeine.cache.LocalAsyncCache$AsyncBulkCompleter accept
WARNING: Exception thrown during asynchronous load
java.util.concurrent.CompletionException: com.starrocks.sql.analyzer.SemanticException: Statistics query fail | Error Message [INTERNAL_ERROR] | QueryId [99898dd2-87af-11ef-acea-52540029b16c] | SQL [SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as int(11))) as string), cast(min(cast(min as int(11))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “item_sold” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as varchar(32))) as string), cast(min(cast(min as varchar(32))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “item_id” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as bigint(20))) as string), cast(min(cast(min as bigint(20))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “cat1” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as varchar(4096))) as string), cast(min(cast(min as varchar(4096))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “pvalue” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as int(11))) as string), cast(min(cast(min as int(11))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “sold” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as double)) as string), cast(min(cast(min as double)) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “price” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as double)) as string), cast(min(cast(min as double)) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “sales” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as varchar(32))) as string), cast(min(cast(min as varchar(32))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “sku_id” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as date)) as string), cast(min(cast(min as date)) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “time” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as varchar(255))) as string), cast(min(cast(min as varchar(255))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “pkey” GROUP BY db_id, table_id, column_name]
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314)
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1702)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.starrocks.sql.analyzer.SemanticException: Statistics query fail | Error Message [INTERNAL_ERROR] | QueryId [99898dd2-87af-11ef-acea-52540029b16c] | SQL [SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as int(11))) as string), cast(min(cast(min as int(11))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “item_sold” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as varchar(32))) as string), cast(min(cast(min as varchar(32))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “item_id” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as bigint(20))) as string), cast(min(cast(min as bigint(20))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “cat1” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as varchar(4096))) as string), cast(min(cast(min as varchar(4096))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “pvalue” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as int(11))) as string), cast(min(cast(min as int(11))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “sold” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as double)) as string), cast(min(cast(min as double)) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “price” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as double)) as string), cast(min(cast(min as double)) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “sales” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as varchar(32))) as string), cast(min(cast(min as varchar(32))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “sku_id” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as date)) as string), cast(min(cast(min as date)) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “time” GROUP BY db_id, table_id, column_name UNION ALL SELECT cast(1 as INT), now(), db_id, table_id, column_name, sum(row_count), cast(sum(data_size) as bigint), hll_union_agg(ndv), sum(null_count), cast(max(cast(max as varchar(255))) as string), cast(min(cast(min as varchar(255))) as string) FROM column_statistics WHERE table_id = 5392891 and column_name = “pkey” GROUP BY db_id, table_id, column_name]
at com.starrocks.statistic.StatisticExecutor.executeDQL(StatisticExecutor.java:270)
at com.starrocks.statistic.StatisticExecutor.executeStatisticDQL(StatisticExecutor.java:253)
at com.starrocks.statistic.StatisticExecutor.queryStatisticSync(StatisticExecutor.java:85)
at com.starrocks.sql.optimizer.statistics.ColumnBasicStatsCacheLoader.queryStatisticsData(ColumnBasicStatsCacheLoader.java:111)
at com.starrocks.sql.optimizer.statistics.ColumnBasicStatsCacheLoader.lambda$asyncLoadAll$1(ColumnBasicStatsCacheLoader.java:77)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
… 3 more
- 慢查询：
- Profile信息
- 并行度：show variables like ‘%parallel_fragment_exec_instance_num%’;
- pipeline是否开启：show variables like ‘%pipeline%’;
- be节点cpu和内存使用率截图
- 查询报错：
- query_dump，怎么获取query_dump文件
- be crash
- be.out
- coredump，如何获取coredump
- 外表查询报错
- be.out和fe.warn.log