【详述】并没有什么任务执行，突然BE挂掉一台，启动起来了，但是一直有报错信息，也无法连接到FE。并没有core文件
【背景】查询了几个小sql
【业务影响】
【StarRocks版本】2.4.1
【集群规模】1fe +3be
【机器信息】8C/32G
【附件】
启动的日志：
W1207 11:35:06.200438 15361 mem_hook.cpp:255] large memory alloc: 1271074463 bytes, stack:
@ 0x368dbeb malloc
@ 0x62a8105 operator new()
@ 0x631f3ba std::__cxx11::basic_string<>::_M_mutate()
@ 0x631fde0 std::__cxx11::basic_string<>::_M_replace_aux()
@ 0x46847a3 rocksdb::CompressBlock()
@ 0x4687797 rocksdb::BlockBasedTableBuilder::CompressAndVerifyBlock()
@ 0x4687ed8 rocksdb::BlockBasedTableBuilder::WriteBlock()
@ 0x468862a rocksdb::BlockBasedTableBuilder::WriteBlock()
@ 0x468abc8 rocksdb::BlockBasedTableBuilder::Finish()
@ 0x46355f2 rocksdb::CompactionJob::FinishCompactionOutputFile()
@ 0x463d16e rocksdb::CompactionJob::ProcessKeyValueCompaction()
@ 0x463e3ac rocksdb::CompactionJob::Run()
@ 0x44087bd rocksdb::DBImpl::BackgroundCompaction()
@ 0x4409e4c rocksdb::DBImpl::BackgroundCallCompaction()
@ 0x440a51b rocksdb::DBImpl::BGWorkCompaction()
@ 0x45fba01 rocksdb::ThreadPoolImpl::Impl::BGThread()
@ 0x45fbc0e rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper()
@ 0x6321d20 execute_native_thread_routine
@ 0x7f5752179ea5 start_thread
@ 0x7f57517948dd __clone
@ (nil) (unknown)
W1207 11:35:30.212355 15361 mem_hook.cpp:255] large memory alloc: 1371810197 bytes, stack:
@ 0x368dbeb malloc
@ 0x62a8105 operator new()
@ 0x631f3ba std::__cxx11::basic_string<>::_M_mutate()
@ 0x63209db std::__cxx11::basic_string<>::_M_append()
@ 0x46941a5 rocksdb::BlockBuilder::Finish()
@ 0x46885dd rocksdb::BlockBasedTableBuilder::WriteBlock()
@ 0x468a99b rocksdb::BlockBasedTableBuilder::Add()
@ 0x463cfb0 rocksdb::CompactionJob::ProcessKeyValueCompaction()
@ 0x463e3ac rocksdb::CompactionJob::Run()
@ 0x44087bd rocksdb::DBImpl::BackgroundCompaction()
@ 0x4409e4c rocksdb::DBImpl::BackgroundCallCompaction()
@ 0x440a51b rocksdb::DBImpl::BGWorkCompaction()
@ 0x45fba01 rocksdb::ThreadPoolImpl::Impl::BGThread()
@ 0x45fbc0e rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper()
@ 0x6321d20 execute_native_thread_routine
@ 0x7f5752179ea5 start_thread
@ 0x7f57517948dd __clone
@ (nil) (unknown)
W1207 11:35:33.867749 15361 mem_hook.cpp:255] large memory alloc: 1086023671 bytes, stack:
@ 0x368dbeb malloc
@ 0x62a8105 operator new()
@ 0x631f3ba std::__cxx11::basic_string<>::_M_mutate()
@ 0x63209db std::__cxx11::basic_string<>::_M_append()
@ 0x46941a5 rocksdb::BlockBuilder::Finish()
@ 0x46885dd rocksdb::BlockBasedTableBuilder::WriteBlock()
@ 0x468abc8 rocksdb::BlockBasedTableBuilder::Finish()
@ 0x46355f2 rocksdb::CompactionJob::FinishCompactionOutputFile()
@ 0x463d16e rocksdb::CompactionJob::ProcessKeyValueCompaction()
@ 0x463e3ac rocksdb::CompactionJob::Run()
@ 0x44087bd rocksdb::DBImpl::BackgroundCompaction()
@ 0x4409e4c rocksdb::DBImpl::BackgroundCallCompaction()
@ 0x440a51b rocksdb::DBImpl::BGWorkCompaction()
@ 0x45fba01 rocksdb::ThreadPoolImpl::Impl::BGThread()
@ 0x45fbc0e rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper()
@ 0x6321d20 execute_native_thread_routine
@ 0x7f5752179ea5 start_thread
@ 0x7f57517948dd __clone
@ (nil) (unknown)
挂掉以后查的日志：
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
E1207 11:24:37.706260 18458 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24745, limit: 1000
W1207 11:24:37.706331 18458 load_channel.cpp:76] Fail to open index 53342 of load adc6a6df75de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24745, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
W1207 11:24:37.797430 18458 internal_service.cpp:186] exec multi plan fragments failed, errmsg=Memory of process exceed limit. Start execute plan fragment. Used: 30896505328, Limit: 28699114843. Mem usage has exceed the limit of BE
E1207 11:24:38.051223 18454 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:38.051314 18454 load_channel.cpp:76] Fail to open index 53342 of load add8a84375de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
E1207 11:24:38.137099 18458 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:38.137171 18458 load_channel.cpp:76] Fail to open index 53342 of load ae086ad575de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
W1207 11:24:38.219334 18452 internal_service.cpp:186] exec multi plan fragments failed, errmsg=Memory of process exceed limit. Start execute plan fragment. Used: 30769692544, Limit: 28699114843. Mem usage has exceed the limit of BE
E1207 11:24:38.257088 18454 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:38.257179 18454 load_channel.cpp:76] Fail to open index 53342 of load ae1a6c3975de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
E1207 11:24:38.340080 18454 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:38.340148 18454 load_channel.cpp:76] Fail to open index 53342 of load ae27648b75de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
W1207 11:24:38.426959 18452 internal_service.cpp:186] exec multi plan fragments failed, errmsg=Memory of process exceed limit. Start execute plan fragment. Used: 30769447328, Limit: 28699114843. Mem usage has exceed the limit of BE
E1207 11:24:38.458727 18452 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:38.458792 18452 load_channel.cpp:76] Fail to open index 53342 of load ae398cff75de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
W1207 11:24:38.336563 18532 mem_hook.cpp:255] large memory alloc: 1887308694 bytes, stack:
@ 0x368e6cb realloc
@ 0x492690d json2pb::PbToJsonConverter::_PbFieldToJson<>()
@ 0x4924dcc json2pb::PbToJsonConverter::Convert<>()
@ 0x4926ad1 json2pb::PbToJsonConverter::_PbFieldToJson<>()
@ 0x4924dcc json2pb::PbToJsonConverter::Convert<>()
@ 0x4924dcc json2pb::PbToJsonConverter::Convert<>()
@ 0x492712d json2pb::ProtoMessageToJsonStream<>()
@ 0x4918a83 json2pb::ProtoMessageToJson()
@ 0x2f1b88b starrocks::TabletMeta::to_json()
@ 0x3bb24ef starrocks::MetaAction::_handle_header()
@ 0x3bb27b1 starrocks::MetaAction::handle()
@ 0x48b05c7 evhttp_handle_request
@ 0x48b1273 bufferevent_readcb
@ 0x489d852 event_process_active_single_queue
@ 0x489df8f event_base_loop
@ 0x3b8d474 _ZZN9starrocks12EvHttpServer5startEvENKUlvE_clEv
@ 0x6321d20 execute_native_thread_routine
@ 0x7f18dfe26ea5 start_thread
@ 0x7f18df4418dd __clone
@ (nil) (unknown)
W1207 11:24:38.436020 18542 mem_hook.cpp:255] large memory alloc: 6369666843 bytes, stack:
@ 0x368e6cb realloc
@ 0x49183b7 butil::rapidjson::PrettyWriter<>::PrettyPrefix()
@ 0x49268ad json2pb::PbToJsonConverter::_PbFieldToJson<>()
@ 0x4924dcc json2pb::PbToJsonConverter::Convert<>()
@ 0x4926ad1 json2pb::PbToJsonConverter::_PbFieldToJson<>()
@ 0x4924dcc json2pb::PbToJsonConverter::Convert<>()
@ 0x4924dcc json2pb::PbToJsonConverter::Convert<>()
@ 0x492712d json2pb::ProtoMessageToJsonStream<>()
@ 0x4918a83 json2pb::ProtoMessageToJson()
@ 0x2f1b88b starrocks::TabletMeta::to_json()
@ 0x3bb24ef starrocks::MetaAction::_handle_header()
@ 0x3bb27b1 starrocks::MetaAction::handle()
@ 0x48b05c7 evhttp_handle_request
@ 0x48b1273 bufferevent_readcb
@ 0x489d852 event_process_active_single_queue
@ 0x489df8f event_base_loop
@ 0x3b8d474 _ZZN9starrocks12EvHttpServer5startEvENKUlvE_clEv
@ 0x6321d20 execute_native_thread_routine
@ 0x7f18dfe26ea5 start_thread
@ 0x7f18df4418dd __clone
@ (nil) (unknown)
E1207 11:24:38.538271 18452 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:38.538334 18452 load_channel.cpp:76] Fail to open index 53342 of load ae459af175de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
W1207 11:24:38.618711 18452 internal_service.cpp:186] exec multi plan fragments failed, errmsg=Memory of process exceed limit. Start execute plan fragment. Used: 31708300464, Limit: 28699114843. Mem usage has exceed the limit of BE
E1207 11:24:38.649870 18452 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:38.649977 18452 load_channel.cpp:76] Fail to open index 53342 of load ae56b1f575de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
E1207 11:24:38.753237 18458 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:38.753350 18458 load_channel.cpp:76] Fail to open index 53342 of load ae66696775de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
W1207 11:24:38.853817 18454 internal_service.cpp:186] exec multi plan fragments failed, errmsg=Memory of process exceed limit. Start execute plan fragment. Used: 31708268976, Limit: 28699114843. Mem usage has exceed the limit of BE
E1207 11:24:38.885825 18452 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:38.885906 18452 load_channel.cpp:76] Fail to open index 53342 of load ae7a8dab75de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
E1207 11:24:39.000069 18454 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:39.000154 18454 load_channel.cpp:76] Fail to open index 53342 of load ae8c19dd75de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
W1207 11:24:38.844578 18067 mem_hook.cpp:255] large memory alloc: 1314512555 bytes, stack:
@ 0x368dbeb malloc
@ 0x62a8105 operator new()
@ 0x631f3ba std::__cxx11::basic_string<>::_M_mutate()
@ 0x63209db std::__cxx11::basic_string<>::_M_append()
@ 0x46941a5 rocksdb::BlockBuilder::Finish()
@ 0x46885dd rocksdb::BlockBasedTableBuilder::WriteBlock()
@ 0x468a99b rocksdb::BlockBasedTableBuilder::Add()
@ 0x461920f rocksdb::BuildTable()
@ 0x4466917 rocksdb::FlushJob::WriteLevel0Table()
@ 0x4468330 rocksdb::FlushJob::Run()
@ 0x43fd135 rocksdb::DBImpl::FlushMemTableToOutputFile()
@ 0x4400756 rocksdb::DBImpl::FlushMemTablesToOutputFiles()
@ 0x440294d rocksdb::DBImpl::BackgroundFlush()
@ 0x4405de9 rocksdb::DBImpl::BackgroundCallFlush()
@ 0x45fba01 rocksdb::ThreadPoolImpl::Impl::BGThread()
@ 0x45fbc0e rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper()
@ 0x6321d20 execute_native_thread_routine
@ 0x7f18dfe26ea5 start_thread
@ 0x7f18df4418dd __clone
@ (nil) (unknown)
W1207 11:24:39.123988 18458 internal_service.cpp:186] exec multi plan fragments failed, errmsg=Memory of process exceed limit. Start execute plan fragment. Used: 33051410600, Limit: 28699114843. Mem usage has exceed the limit of BE
E1207 11:24:39.171751 18454 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:39.171916 18454 load_channel.cpp:76] Fail to open index 53342 of load aea6319175de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
E1207 11:24:39.285120 18454 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:39.285192 18454 load_channel.cpp:76] Fail to open index 53342 of load aeb796b375de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
W1207 11:24:39.377166 18454 internal_service.cpp:186] exec multi plan fragments failed, errmsg=Memory of process exceed limit. Start execute plan fragment. Used: 33051485192, Limit: 28699114843. Mem usage has exceed the limit of BE
E1207 11:24:39.405414 18454 delta_writer.cpp:110] Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
W1207 11:24:39.405489 18454 load_channel.cpp:76] Fail to open index 53342 of load aeca0d4775de11ed-9055fa163eb174ee: Service unavailable: Too many versions. tablet_id: 88973, version_count: 24834, limit: 1000
/root/starrocks/be/src/storage/delta_writer.cpp:22 writer->_init()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:482 res.status()
/root/starrocks/be/src/runtime/local_tablets_channel.cpp:228 _open_all_writers(params)
监控截图：
