【详述】Starrocks BE节点突然挂掉了,重启后几小时后又会挂掉,每次挂掉的节点不一样
【背景】停机升级过内存,升级之后就开始报错了
【业务影响】
【是否存算分离】
【StarRocks版本】例如:3.1.17
【集群规模】例如:3fe(1 leader+2follower)+4be
【机器信息】CPU虚拟核/内存/网卡,例如:16C/32G/百兆
【联系方式】287935361@qq.com
【附件】be.out (21.7 KB)
start time: Fri Dec 19 16:38:45 CST 2025, server uptime: 16:38:45 up 2:00, 1 user, load average: 2.85, 9.98, 7.61
3.1.17 RELEASE (build 67ae3b7)
query_id:22ce96e1-dcb9-11f0-8e7d-005056bc48cd, fragment_instance:22ce96e1-dcb9-11f0-8e7d-005056bc48d0
tracker:process consumption: 36727084989
tracker:query_pool consumption: 333701936
tracker:load consumption: 2518122882
tracker:metadata consumption: 8281192270
tracker:tablet_metadata consumption: 2410609223
tracker:rowset_metadata consumption: 2462970692
tracker:segment_metadata consumption: 521465796
tracker:column_metadata consumption: 2886146559
tracker:tablet_schema consumption: 1296915143
tracker:segment_zonemap consumption: 445433752
tracker:short_key_index consumption: 6028521
tracker:column_zonemap_index consumption: 861041111
tracker:ordinal_index consumption: 887829136
tracker:bitmap_index consumption: 0
tracker:bloom_filter_index consumption: 744952
tracker:compaction consumption: 0
tracker:schema_change consumption: 0
tracker:column_pool consumption: 1576008084
tracker:page_cache consumption: 10947442848
tracker:update consumption: 5654380588
tracker:chunk_allocator consumption: 893891704
tracker:clone consumption: 0
tracker:consistency consumption: 0
tracker:datacache consumption: 0
tracker:replication consumption: 0
*** Aborted at 1766134815 (unix time) try "date -d @1766134815" if you are using GNU date ***
PC: @ 0x5771edf starrocks::ScalarColumnIterator::init()
*** SIGSEGV (@0x44) received by PID 92330 (TID 0x14676c7f4640) from PID 68; stack trace: ***
@ 0x688b642 google::(anonymous namespace)::FailureSignalHandler()
@ 0x146a15e3fc30 (unknown)
@ 0x5771edf starrocks::ScalarColumnIterator::init()
@ 0x5283383 starrocks::SegmentIterator::_init_column_iterator_by_cid()
@ 0x529767b starrocks::SegmentIterator::_init_column_iterators<>()
@ 0x528a502 starrocks::SegmentIterator::_init()
@ 0x528a9c0 starrocks::SegmentIterator::do_get_next()
@ 0x5994675 starrocks::SegmentIteratorWrapper::do_get_next()
@ 0x57c62d3 starrocks::TimedChunkIterator::do_get_next()
@ 0x535125c starrocks::UnionIterator::do_get_next()
@ 0x5341706 starrocks::TabletReader::do_get_next()
@ 0x3b0271b starrocks::pipeline::OlapChunkSource::_read_chunk_from_storage()
@ 0x3b02e42 starrocks::pipeline::OlapChunkSource::_read_chunk()
@ 0x3afba17 starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking()
@ 0x37c0c38 _ZZN9starrocks8pipeline12ScanOperator18_trigger_next_scanEPNS_12RuntimeStateEiENKUlvE_clEv
@ 0x38d4c91 starrocks::workgroup::ScanExecutor::worker_thread()
@ 0x2ed30ec starrocks::ThreadPool::dispatch_thread()
@ 0x2ecc7ba starrocks::Thread::supervise_thread()
@ 0x146a15e8b2ea start_thread
@ 0x146a15f103c0 __clone3
@ 0x0 (unknown)
start time: Fri Dec 19 17:03:44 CST 2025, server uptime: 17:03:44 up 2:25, 1 user, load average: 2.53, 10.54, 8.55
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/starrocks/be/lib/jni-packages/starrocks-jdbc-bridge-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/starrocks/be/lib/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
3.1.17 RELEASE (build 67ae3b7)
query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000
tracker:process consumption: 25841128144
tracker:query_pool consumption: 0
tracker:load consumption: 0
tracker:metadata consumption: 6872690218
tracker:tablet_metadata consumption: 2420297505
tracker:rowset_metadata consumption: 2008597179
tracker:segment_metadata consumption: 320664273
tracker:column_metadata consumption: 2123131261
tracker:tablet_schema consumption: 1296837793
tracker:segment_zonemap consumption: 246950622
tracker:short_key_index consumption: 10532075
tracker:column_zonemap_index consumption: 475653301
tracker:ordinal_index consumption: 680613504
tracker:bitmap_index consumption: 0
tracker:bloom_filter_index consumption: 598328
tracker:compaction consumption: 0
tracker:schema_change consumption: 0
tracker:column_pool consumption: 1577860004
tracker:page_cache consumption: 10923907584
tracker:update consumption: 195672700
tracker:chunk_allocator consumption: 1382911912
tracker:clone consumption: 0
tracker:consistency consumption: 0
tracker:datacache consumption: 0
tracker:replication consumption: 0
*** Aborted at 1766179060 (unix time) try "date -d @1766179060" if you are using GNU date ***
PC: @ 0x5771edf starrocks::ScalarColumnIterator::init()
*** SIGSEGV (@0x44) received by PID 107702 (TID 0x14cff37d0640) from PID 68; stack trace: ***
@ 0x688b642 google::(anonymous namespace)::FailureSignalHandler()
@ 0x14d2aba7d1bd os::Linux::chained_handler()
@ 0x14d2aba82eff JVM_handle_linux_signal
@ 0x14d2aba74b28 signalHandler()
@ 0x14d2aaa3fc30 (unknown)
@ 0x5771edf starrocks::ScalarColumnIterator::init()
@ 0x5283383 starrocks::SegmentIterator::_init_column_iterator_by_cid()
@ 0x529767b starrocks::SegmentIterator::_init_column_iterators<>()
@ 0x528a502 starrocks::SegmentIterator::_init()
@ 0x528ad64 starrocks::SegmentIterator::do_get_next()
@ 0x529b795 starrocks::SegmentIterator::do_get_next()
@ 0x537476e starrocks::MergeEntry<>::next()
@ 0x53786db starrocks::RowsetMergerImpl<>::_do_merge_horizontally()
@ 0x537a37d starrocks::RowsetMergerImpl<>::_do_merge_vertically()
@ 0x537c001 starrocks::RowsetMergerImpl<>::do_merge()
@ 0x536d20f starrocks::compaction_merge_rowsets()
@ 0x5228c3d starrocks::TabletUpdates::_do_compaction()
@ 0x522ad2c starrocks::TabletUpdates::compaction_for_size_tiered()
@ 0x522b7ea starrocks::TabletUpdates::compaction()
@ 0x516c70b starrocks::StorageEngine::_perform_update_compaction()
@ 0x50fe7e4 starrocks::StorageEngine::_update_compaction_thread_callback()
@ 0x8cca900 execute_native_thread_routine
@ 0x14d2aaa8b2ea start_thread
@ 0x14d2aab103c0 __clone3
@ 0x0 (unknown)
尝试升级到3.5 最新版试试