fe节点异常停止

为了更快的定位您的问题,请提供以下信息,谢谢
【详述】fe宕机
【背景】drop了一个be节点
【业务影响】
【是否存算分离】否
【StarRocks版本】例如:3.1.4
【集群规模】例如:3fe(3 )+12be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,例如:48C/64G/万兆
【联系方式】18840044216
【附件】

2023-11-27 21:18:19,345 INFO (starrocks-mysql-nio-pool-3441|4001571) [QeProcessorImpl.registerQuery():100] register query id = 6e931d6b-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,346 INFO (starrocks-mysql-nio-pool-3443|4001573) [QeProcessorImpl.registerQuery():100] register query id = 6e93447c-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,347 INFO (starrocks-mysql-nio-pool-3447|4001577) [QeProcessorImpl.registerQuery():100] register query id = 6e936b8d-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,356 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6e94a415-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,358 INFO (starrocks-mysql-nio I/O-2|168) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41870, connectionId=31195
2023-11-27 21:18:19,358 INFO (starrocks-mysql-nio-pool-3447|4001577) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001577. remote=/10.0.1.211:41870, connectionId=31195
2023-11-27 21:18:19,359 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e93447c-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,359 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e931d6b-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,360 INFO (starrocks-mysql-nio I/O-2|168) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41874, connectionId=31196
2023-11-27 21:18:19,360 INFO (starrocks-mysql-nio-pool-3443|4001573) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001573. remote=/10.0.1.211:41874, connectionId=31196
2023-11-27 21:18:19,360 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e936b8d-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,362 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e94a415-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,362 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6e95b58b-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,363 INFO (starrocks-mysql-nio-pool-3443|4001573) [QeProcessorImpl.registerQuery():100] register query id = 6e9603ac-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,364 INFO (starrocks-mysql-nio-pool-3447|4001577) [QeProcessorImpl.registerQuery():100] register query id = 6e9603ad-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,364 INFO (starrocks-mysql-nio I/O-2|168) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41880, connectionId=31197
2023-11-27 21:18:19,364 INFO (starrocks-mysql-nio-pool-3450|4001581) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001581. remote=/10.0.1.211:41880, connectionId=31197
2023-11-27 21:18:19,367 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e95b58b-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,367 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9603ac-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,367 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9603ad-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,370 INFO (nioEventLoopGroup-6-13|303) [RestBaseAction.handleRequest():70] receive http request. url=/api/bootstrap?cluster_id=1531098948&token=85a39bb5-d5cd-479a-9e5f-7a6676a88eae
2023-11-27 21:18:19,371 INFO (starrocks-mysql-nio-pool-3447|4001577) [QeProcessorImpl.registerQuery():100] register query id = 6e971524-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,373 INFO (starrocks-mysql-nio I/O-1|167) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41888, connectionId=31198
2023-11-27 21:18:19,373 INFO (starrocks-mysql-nio-pool-3447|4001577) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001577. remote=/10.0.1.211:41888, connectionId=31198
2023-11-27 21:18:19,375 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e971524-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,376 INFO (starrocks-mysql-nio I/O-2|168) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41890, connectionId=31199
2023-11-27 21:18:19,376 INFO (starrocks-mysql-nio-pool-3450|4001581) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001581. remote=/10.0.1.211:41890, connectionId=31199
2023-11-27 21:18:19,377 INFO (starrocks-mysql-nio-pool-3447|4001577) [QeProcessorImpl.registerQuery():100] register query id = 6e98269c-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,380 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6e9874be-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,381 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e98269c-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,382 INFO (starrocks-mysql-nio I/O-2|168) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41892, connectionId=31200
2023-11-27 21:18:19,382 INFO (starrocks-mysql-nio-pool-3450|4001581) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001581. remote=/10.0.1.211:41892, connectionId=31200
2023-11-27 21:18:19,384 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9874be-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,389 INFO (starrocks-mysql-nio I/O-4|170) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41894, connectionId=31201
2023-11-27 21:18:19,389 INFO (starrocks-mysql-nio-pool-3450|4001581) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001581. remote=/10.0.1.211:41894, connectionId=31201
2023-11-27 21:18:19,389 INFO (starrocks-mysql-nio I/O-3|169) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41896, connectionId=31202
2023-11-27 21:18:19,389 INFO (starrocks-mysql-nio-pool-3441|4001571) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001571. remote=/10.0.1.211:41896, connectionId=31202
2023-11-27 21:18:19,390 INFO (starrocks-mysql-nio-pool-3443|4001573) [QeProcessorImpl.registerQuery():100] register query id = 6e99d459-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,393 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6e9a709b-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,394 INFO (starrocks-mysql-nio I/O-3|169) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41898, connectionId=31203
2023-11-27 21:18:19,394 INFO (starrocks-mysql-nio-pool-3443|4001573) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001573. remote=/10.0.1.211:41898, connectionId=31203
2023-11-27 21:18:19,395 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6e9ae5cc-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,396 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e99d459-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,396 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9a709b-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,396 INFO (starrocks-mysql-nio-pool-3443|4001573) [QeProcessorImpl.registerQuery():100] register query id = 6e9b0cdd-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,398 INFO (starrocks-mysql-nio-pool-3441|4001571) [QeProcessorImpl.registerQuery():100] register query id = 6e9b33ef-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,400 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9ae5cc-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,402 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9b33ef-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,402 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9b0cdd-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,405 INFO (starrocks-mysql-nio I/O-1|167) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41900, connectionId=31204
2023-11-27 21:18:19,405 INFO (starrocks-mysql-nio-pool-3441|4001571) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001571. remote=/10.0.1.211:41900, connectionId=31204
2023-11-27 21:18:19,409 INFO (starrocks-mysql-nio-pool-3441|4001571) [QeProcessorImpl.registerQuery():100] register query id = 6e9ce1ad-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,410 INFO (starrocks-mysql-nio I/O-2|168) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41902, connectionId=31205
2023-11-27 21:18:19,410 INFO (starrocks-mysql-nio-pool-3443|4001573) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001573. remote=/10.0.1.211:41902, connectionId=31205
2023-11-27 21:18:19,412 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9ce1ad-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,414 INFO (starrocks-mysql-nio-pool-3443|4001573) [QeProcessorImpl.registerQuery():100] register query id = 6e9dcc11-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,414 INFO (starrocks-mysql-nio I/O-3|169) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41904, connectionId=31206
2023-11-27 21:18:19,414 INFO (starrocks-mysql-nio-pool-3441|4001571) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001571. remote=/10.0.1.211:41904, connectionId=31206
2023-11-27 21:18:19,415 INFO (starrocks-mysql-nio I/O-1|167) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41906, connectionId=31207
2023-11-27 21:18:19,415 INFO (starrocks-mysql-nio-pool-3450|4001581) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001581. remote=/10.0.1.211:41906, connectionId=31207
2023-11-27 21:18:19,418 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9dcc11-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,420 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6e9eb677-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,421 INFO (starrocks-mysql-nio I/O-1|167) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41908, connectionId=31208
2023-11-27 21:18:19,421 INFO (starrocks-mysql-nio-pool-3441|4001571) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001571. remote=/10.0.1.211:41908, connectionId=31208
2023-11-27 21:18:19,421 INFO (starrocks-mysql-nio-pool-3443|4001573) [QeProcessorImpl.registerQuery():100] register query id = 6e9eb678-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,424 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9eb677-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,425 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6e9f52bb-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,426 INFO (starrocks-mysql-nio I/O-1|167) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41910, connectionId=31209
2023-11-27 21:18:19,426 INFO (starrocks-mysql-nio-pool-3443|4001573) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001573. remote=/10.0.1.211:41910, connectionId=31209
2023-11-27 21:18:19,427 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9eb678-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,428 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6e9feefd-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,428 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9f52bb-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,430 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6ea03d20-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,432 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e9feefd-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,437 INFO (starrocks-mysql-nio I/O-1|167) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41912, connectionId=31210
2023-11-27 21:18:19,437 INFO (starrocks-mysql-nio-pool-3457|4001590) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001590. remote=/10.0.1.211:41912, connectionId=31210
2023-11-27 21:18:19,438 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ea03d20-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,439 INFO (starrocks-mysql-nio I/O-2|168) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41914, connectionId=31211
2023-11-27 21:18:19,439 INFO (starrocks-mysql-nio-pool-3457|4001590) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001590. remote=/10.0.1.211:41914, connectionId=31211
2023-11-27 21:18:19,441 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6ea1c3ca-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,444 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ea1c3ca-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,445 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6ea2871f-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,446 INFO (starrocks-mysql-nio I/O-4|170) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41916, connectionId=31212
2023-11-27 21:18:19,446 INFO (starrocks-mysql-nio-pool-3443|4001573) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001573. remote=/10.0.1.211:41916, connectionId=31212
2023-11-27 21:18:19,447 INFO (starrocks-mysql-nio I/O-1|167) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41918, connectionId=31213
2023-11-27 21:18:19,447 INFO (starrocks-mysql-nio-pool-3457|4001590) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001590. remote=/10.0.1.211:41918, connectionId=31213
2023-11-27 21:18:19,449 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ea2871f-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,451 INFO (starrocks-mysql-nio-pool-3443|4001573) [QeProcessorImpl.registerQuery():100] register query id = 6ea34a74-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,453 INFO (starrocks-mysql-nio-pool-3457|4001590) [QeProcessorImpl.registerQuery():100] register query id = 6ea39896-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,456 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ea34a74-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,457 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ea39896-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,458 INFO (starrocks-mysql-nio-pool-3457|4001590) [QeProcessorImpl.registerQuery():100] register query id = 6ea482fc-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,462 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ea482fc-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,464 INFO (starrocks-mysql-nio I/O-4|170) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41920, connectionId=31214
2023-11-27 21:18:19,464 INFO (starrocks-mysql-nio-pool-3457|4001590) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001590. remote=/10.0.1.211:41920, connectionId=31214
2023-11-27 21:18:19,470 INFO (starrocks-mysql-nio-pool-3457|4001590) [QeProcessorImpl.registerQuery():100] register query id = 6ea630b2-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,474 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ea630b2-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,478 INFO (starrocks-mysql-nio I/O-4|170) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41922, connectionId=31215
2023-11-27 21:18:19,478 INFO (starrocks-mysql-nio-pool-3457|4001590) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001590. remote=/10.0.1.211:41922, connectionId=31215
2023-11-27 21:18:19,485 INFO (starrocks-mysql-nio-pool-3457|4001590) [QeProcessorImpl.registerQuery():100] register query id = 6ea85398-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,488 INFO (starrocks-mysql-nio I/O-1|167) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41924, connectionId=31216
2023-11-27 21:18:19,488 INFO (starrocks-mysql-nio-pool-3457|4001590) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001590. remote=/10.0.1.211:41924, connectionId=31216
2023-11-27 21:18:19,489 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ea85398-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,491 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6ea96509-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,494 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6ea9da3a-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,495 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ea96509-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,497 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ea9da3a-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,512 INFO (starrocks-mysql-nio I/O-1|167) [AcceptListener.handleEvent():79] Connection established. remote=/10.0.1.211:41926, connectionId=31217
2023-11-27 21:18:19,512 INFO (starrocks-mysql-nio-pool-3450|4001581) [AcceptListener.lambda$handleEvent$1():87] Connection scheduled to worker thread 4001581. remote=/10.0.1.211:41926, connectionId=31217
2023-11-27 21:18:19,518 INFO (starrocks-mysql-nio-pool-3450|4001581) [QeProcessorImpl.registerQuery():100] register query id = 6ead83c3-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,522 INFO (profile-worker-1|3716) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6ead83c3-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,528 INFO (profile-worker-0|3226) [QeProcessorImpl.unregisterQuery():139] deregister query id = 6e87d35e-8d27-11ee-a7a2-00163e13470f
2023-11-27 21:18:19,717 INFO (audit loader thread|142) [StarrocksStreamLoader.loadBatch():143] AuditLoader plugin load with label: audit_20231127_211819_172_16_4_109_9010, response code: 200, msg: OK, content: { “TxnId”: 66825936, “Label”: “audit_20231127_211819_172_16_4_109_9010”, “Status”: “Fail”, “Message”: “transaction not found: 66825936”, “NumberTotalRows”: 13, “NumberLoadedRows”: 13, “NumberFilteredRows”: 0, “NumberUnselectedRows”: 0, “LoadBytes”: 5620, “LoadTimeMs”: 281, “BeginTxnTimeMs”: 36, “StreamLoadPlanTimeMs”: 5, “ReadDataTimeMs”: 0, “WriteDataTimeMs”: 76, “CommitAndPublishTimeMs”: 0}
2023-11-27 21:18:20,261 WARN (MASTER 172.16.4.109_9010_1700813317245(5)|127) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN
2023-11-27 21:18:20,261 INFO (stateChangeExecutor|94) [StateChangeExecutor.runOneCycle():85] begin to transfer FE type from LEADER to UNKNOWN
2023-11-27 21:18:20,261 ERROR (stateChangeExecutor|94) [StateChangeExecutor.runOneCycle():171] transfer FE type from LEADER to UNKNOWN. exit
2023-11-27 21:18:20,261 ERROR (heartbeat mgr|22) [Daemon.run():117] daemon thread got exception. name: heartbeat mgr
com.sleepycat.je.rep.UnknownMasterException: java.lang.Exception: Stacktrace where exception below was rethrown (com.sleepycat.je.rep.UnknownMasterException)
at com.sleepycat.je.DatabaseException.addRethrownStackTrace(DatabaseException.java:139)
at com.sleepycat.je.rep.utilint.RepUtils.addLocalStackTrace(RepUtils.java:408)
at com.sleepycat.je.rep.utilint.RepUtils.access$300(RepUtils.java:54)
at com.sleepycat.je.rep.utilint.RepUtils$ExceptionAwareCountDownLatch.awaitOrException(RepUtils.java:297)
at com.sleepycat.je.rep.utilint.SizeAwaitMap.sizeAwait(SizeAwaitMap.java:128)
at com.sleepycat.je.rep.impl.node.FeederManager.awaitFeederReplicaConnections(FeederManager.java:841)
at com.sleepycat.je.rep.impl.node.DurabilityQuorum.ensureReplicasForCommit(DurabilityQuorum.java:88)
at com.sleepycat.je.rep.impl.RepImpl.txnBeginHook(RepImpl.java:1171)
at com.sleepycat.je.rep.txn.MasterTxn.txnBeginHook(MasterTxn.java:195)
at com.sleepycat.je.txn.Txn.initTxn(Txn.java:384)
at com.sleepycat.je.txn.Txn.(Txn.java:288)
at com.sleepycat.je.txn.Txn.(Txn.java:267)
at com.sleepycat.je.rep.txn.MasterTxn.(MasterTxn.java:146)
at com.sleepycat.je.rep.txn.MasterTxn$1.create(MasterTxn.java:117)
at com.sleepycat.je.rep.txn.MasterTxn.create(MasterTxn.java:435)
at com.sleepycat.je.rep.impl.RepImpl.createRepUserTxn(RepImpl.java:1145)
at com.sleepycat.je.txn.Txn.createAutoTxn(Txn.java:334)
at com.sleepycat.je.txn.LockerFactory.getWritableLocker(LockerFactory.java:79)
at com.sleepycat.je.Environment.setupDatabase(Environment.java:816)
at com.sleepycat.je.Environment.openDatabase(Environment.java:668)
at com.starrocks.journal.bdbje.BDBEnvironment.openDatabase(BDBEnvironment.java:444)
at com.starrocks.journal.bdbje.BDBJEJournal.getMaxJournalId(BDBJEJournal.java:155)
at com.starrocks.server.GlobalStateMgr.getMaxJournalId(GlobalStateMgr.java:3189)
at com.starrocks.system.HeartbeatMgr.runAfterCatalogReady(HeartbeatMgr.java:178)
at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:72)
at com.starrocks.common.util.Daemon.run(Daemon.java:115)

(JE 18.3.16) Node 172.16.4.109_9010_1700813317245(5):/opt/module/doris-meta/bdb is not a master anymore Originally thrown by HA thread: MASTER 172.16.4.109_9010_1700813317245(5)
at com.sleepycat.je.rep.impl.node.FeederManager.runFeeders(FeederManager.java:701) ~[starrocks-bdb-je-18.3.16.jar:?]
at com.sleepycat.je.rep.impl.node.RepNode.run(RepNode.java:1841) ~[starrocks-bdb-je-18.3.16.jar:?]
2023-11-27 21:18:20,264 INFO (Thread-73|175) [StarRocksFE.lambda$addShutdownHook$1():390] start to execute shutdown hook
2023-11-27 21:18:20,267 INFO (Thread-73|175) [StarRocksFE.lambda$addShutdownHook$1():415] shutdown hook end
2023-11-27 22:32:34,499 INFO (main|1) [StarRocksFE.start():129] StarRocks FE starting, version: 3.1.4-0c4b2a3
2023-11-27 22:32:34,511 INFO (main|1) [FrontendOptions.analyzePriorityCidrs():299] configured prior_cidrs value: 172.16.4.109
2023-11-27 22:32:34,514 INFO (main|1) [FrontendOptions.initAddrUseIp():249] Use IP init local addr, IP: /172.16.4.109

fe.gc.log.20231127-223232 (85.9 KB) gc的log

fe.gc.log.20231124-160855 (188.1 KB)

贴下fe.log和meta/bdb/je.info.0

je.info.0 (2.3 MB)

BEINFO.txt (2.2 MB)

我们正在做drop,be节点,有很多的unhealthy的tablet正在修复

大概昨晚9点18分挂的,10点多我给起来的

老师麻烦帮忙看下,今天这又挂了

fullgc导致,把jvm调大吧

老师 我们原来leader是一个32G的机器,放的jvm18g一直没事,最近换了一台64G的当leader且给他jvm设置的50g开始频繁挂,一天挂一次,这个是什么原因啊

可以在内存高的时候获取下jmap -histo pid,看下是哪块占用多,具体参考
[问题排查]内存相关

可以换成 g1 算法试试