【详述】问题详细描述
【背景】做过哪些操作?
【业务影响】
【StarRocks版本】例如:2.3
【集群规模】例如:3fe(1 follower+2observer)+10be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:60C/256G/万兆
【附件】
- fe.warn.log/be.warn.log/相应截图
- 慢查询:
- Profile信息
- 并行度:show variables like ‘%parallel_fragment_exec_instance_num%’;
- cbo是否开启:show variables like ‘%cbo%’;
2022-11-10 12:21:04,064 WARN (thrift-server-pool-124|530) [MasterImpl.finishTask():167] finish task reports bad. request: TFinishTaskRequest(backend:TBackend(host:10.20.52.196, be_port:9060, http_port:8040), task_t
ype:CLONE, signature:2957125, task_status:TStatus(status_code:RUNTIME_ERROR, error_msgs:[Internal error: load snapshot failed, tablet updates is in error state: tablet:2957125 actual row size changed after compacti
on 1781369 -> 1781477 tablet:2957125 #version:2 [891 891.1@1 891.1] pending: rowsets:2
1777 [seg:1 row:1781473 del:0 bytes:79144826 compaction:-45590394]
1778 [seg:1 row:4 del:0 bytes:4718 compaction:33554432]
/root/starrocks/be/src/storage/task/engine_clone_task.cpp:810 tablet->updates()->load_snapshot(snapshot_meta), clone failed.]))
2022-11-10 12:21:07,180 WARN (tablet scheduler|46) [TabletScheduler.runAfterCatalogReady():317] select balance tablets cost too much time: 2 seconds
2022-11-10 12:21:28,185 WARN (tablet scheduler|46) [TabletScheduler.runAfterCatalogReady():317] select balance tablets cost too much time: 2 seconds
2022-11-10 12:21:48,130 WARN (tablet scheduler|46) [TabletScheduler.runAfterCatalogReady():317] select balance tablets cost too much time: 1 seconds
2022-11-10 12:22:17,390 WARN (tablet scheduler|46) [TabletScheduler.runAfterCatalogReady():317] select balance tablets cost too much time: 5 seconds
2022-11-10 12:22:36,023 WARN (tablet scheduler|46) [TabletScheduler.runAfterCatalogReady():317] select balance tablets cost too much time: 3 seconds
2022-11-10 12:22:56,642 WARN (thrift-server-pool-145|552) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 25324119, signature: 2651373
2022-11-10 12:22:56,644 WARN (thrift-server-pool-143|550) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 25324120, signature: 2651373
2022-11-10 12:22:56,644 WARN (thrift-server-pool-123|529) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 25324121, signature: 2651373
2022-11-10 12:22:57,164 WARN (thrift-server-pool-118|524) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 10008, signature: 2651373
2022-11-10 12:22:57,165 WARN (thrift-server-pool-105|502) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 10007, signature: 2651373
2022-11-10 12:22:57,165 WARN (thrift-server-pool-117|523) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 852425, signature: 2651373
2022-11-10 12:22:58,207 WARN (tablet scheduler|46) [TabletScheduler.runAfterCatalogReady():317] select balance tablets cost too much time: 5 seconds
2022-11-10 12:23:04,030 WARN (thrift-server-pool-138|544) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 25324118, signature: 2651378
2022-11-10 12:23:04,030 WARN (thrift-server-pool-102|499) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 852399, signature: 2651378
2022-11-10 12:23:04,030 WARN (thrift-server-pool-104|501) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 10003, signature: 2651378
2022-11-10 12:23:04,030 WARN (thrift-server-pool-129|535) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 25324117, signature: 2651378
2022-11-10 12:23:04,045 WARN (thrift-server-pool-115|521) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 10003, signature: 2651379
2022-11-10 12:23:04,045 WARN (thrift-server-pool-110|516) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 10003, signature: 2651380
2022-11-10 12:23:04,045 WARN (thrift-server-pool-107|512) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 10003, signature: 2651381
2022-11-10 12:23:04,057 WARN (thrift-server-pool-116|522) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 852399, signature: 2651379
2022-11-10 12:23:04,058 WARN (thrift-server-pool-111|517) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 852399, signature: 2651380
2022-11-10 12:23:04,058 WARN (thrift-server-pool-119|525) [MasterImpl.finishTask():194] cannot find task. type: PUBLISH_VERSION, backendId: 852399, signature: 2651381
2022-11-10 12:23:20,180 WARN (tablet scheduler|46) [TabletScheduler.runAfterCatalogReady():317] select balance tablets cost too much time: 5 seconds
2022-11-10 12:23:42,173 WARN (tablet scheduler|46) [Database.readLock():134] slow read lock db:2872030 default_cluster:ea_dw 6686ms
java.lang.Exception: null
at com.starrocks.catalog.Database.readLock(Database.java:134) [starrocks-fe.jar:?]
at com.starrocks.clone.DiskAndTabletLoadReBalancer.getPartitionStats(DiskAndTabletLoadReBalancer.java:1383) [starrocks-fe.jar:?]
at com.starrocks.clone.DiskAndTabletLoadReBalancer.balanceTablet(DiskAndTabletLoadReBalancer.java:1013) [starrocks-fe.jar:?]
at com.starrocks.clone.DiskAndTabletLoadReBalancer.balanceClusterTablet(DiskAndTabletLoadReBalancer.java:949) [starrocks-fe.jar:?]
at com.starrocks.clone.DiskAndTabletLoadReBalancer.selectAlternativeTabletsForCluster(DiskAndTabletLoadReBalancer.java:78) [starrocks-fe.jar:?]
at com.starrocks.clone.Rebalancer.selectAlternativeTablets(Rebalancer.java:63) [starrocks-fe.jar:?]
at com.starrocks.clone.TabletScheduler.selectTabletsForBalance(TabletScheduler.java:1139) [starrocks-fe.jar:?]
at com.starrocks.clone.TabletScheduler.runAfterCatalogReady(TabletScheduler.java:314) [starrocks-fe.jar:?]
at com.starrocks.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:61) [starrocks-fe.jar:?]