【详述】升级4.0.6 explain sql测试,hive外表包含大量分区时,4.0.6 explain耗时超120s,3.4耗时2s,LIST_FS_PARTITIONS阶段耗时异常
【StarRocks版本】4.0.6
异常信息:
err=StarRocksPlannerException: StarRocks planner use long time 123142 ms in memo phase, This probably because 1. FE Full GC, current query allocated=1.2 GiB, optimizerContextInUse=48.2 MiB2. Hive external table fetch metadata took a long time, 3. The SQL is very complex. You could 1. adjust FE JVM config, 2. try query again, 3. enlarge new_planner_optimize_timeout session variable, profile=Planner:
- -- Parser[1] 4ms
- -- Total[1] 2m3s
- -- Analyzer[1] 66ms
- -- Lock[1] 0
- -- AnalyzeDatabase[3] 15ms
- -- HMS.getDatabase[2] 14ms
- -- AnalyzeTemporaryTable[3] 0
- -- AnalyzeTable[3] 48ms
- -- HMS.getTable[2] 48ms
- -- Transformer[1] 20ms
- -- Optimizer[1] 2m3s
- -- MVPreprocess[1] 0
- -- MVTextRewrite[1] 0
- -- RuleBaseOptimize[1] 1m59s
- -- HMS.listPartitionNames[2] 69ms
- -- HMS.getPartitionsByNames[3] 511ms
- -- HMS.getTable[3] 65ms
- -- HMS.getPartitionColumnStats[2] 309ms
- -- HMS.PARTITIONS.LIST_FS_PARTITIONS[46] 1m57s
- -- CostBaseOptimize[1] 3s874ms
- -- HMS.PARTITIONS.LIST_FS_PARTITIONS[1] 3s535ms
HMS:
PARTITIONS:
- LIST_FS_PARTITIONS: 103541
getPartitionColumnStats:
- put_rt: 2203 partitionStats
- ad_daily: 189 partitionStats
getPartitionsByNames:
- put_rt: 2203 partitions
- ad_daily: 189 partitions
Reason: