常见 Crash / BUG / 优化查询

trueeyu · 2024年06月4日 02:01

Join reorder + 窗口函数导致 crash

*** Aborted at 1697526858 (unix time) try "date -d @1697526858" if you are using GNU date ***
PC: @     0x7f0f33393387 __GI_raise
*** SIGABRT (@0x2431) received by PID 9265 (TID 0x7f0e5251a700) from PID 9265; stack trace: ***
    @          0x5960be2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f0f33e48630 (unknown)
    @     0x7f0f33393387 __GI_raise
    @     0x7f0f33394a78 __GI_abort
    @          0x2c8343e starrocks::failure_function()
    @          0x59545bd google::LogMessage::Fail()
    @          0x5956a2f google::LogMessage::SendToLog()
    @          0x595410e google::LogMessage::Flush()
    @          0x5957039 google::LogMessageFatal::~LogMessageFatal()
    @          0x4ea2dc2 _ZN9starrocks20type_dispatch_columnINS_10vectorized13ColumnBuilderEJNS_14TypeDescriptorEmEEEDaNS_13PrimitiveTypeET_DpT0_
    @          0x4ea028b starrocks::vectorized::ColumnHelper::create_column()
    @          0x51cecf0 starrocks::serde::ProtobufChunkDeserializer::deserialize()
    @          0x487e470 starrocks::DataStreamRecvr::SenderQueue::_deserialize_chunk()
    @          0x488178a starrocks::DataStreamRecvr::PipelineSenderQueue::get_chunk()
    @          0x47f9e03 starrocks::DataStreamRecvr::get_chunk_for_pipeline()
    @          0x2fb035a starrocks::pipeline::ExchangeSourceOperator::pull_chunk()
    @          0x2d1dc30 starrocks::pipeline::PipelineDriver::process()
    @          0x4f91993 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x4983a52 starrocks::ThreadPool::dispatch_thread()
    @          0x497e54a starrocks::Thread::supervise_thread()
    @     0x7f0f33e40ea5 start_thread
    @     0x7f0f3345bb0d __clone
    @                0x0 (unknown)

或

build chunk meta error

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/33671
Jira
问题版本:
- 2.5.0 ~ 2.5.14
修复版本:
- 2.5.15+
问题原因:
临时解决办法:
- set global disable_join_reorder=true;

U_1684922297665_5078 · 2023年11月10日 10:07

请问专家，这个bug只能升级版本才能再用起来嘛，现在3.0.1重启FE启动不来了

llppff · 2023年11月16日 04:21

3.0.7和3.1.4已修复

trueeyu · 2024年02月27日 09:00

smp_call_function_many 占用大量CPU

而且有疑似死锁现象

问题版本
- 所有使用了 jemalloc 的版本: >= 2.4
解决办法
- 修改 be/bin/start_backend.sh 将 muzzy_decay_ms:5000,dirty_decay_ms:5000 改为 muzzy_decay_ms:30000,dirty_decay_ms:30000
- 这个参数修改只能缓解，不能根治，需要系统研究下，再解决
- 在Linux中关闭numa，并重启机器

trueeyu · 2023年11月28日 11:09

collect_query_statistics crash

收集 Query 执行后的一些信息时 (如ScanBytes等） crash

*** Aborted at 1698221708 (unix time) try "date -d @1698221708" if you are using GNU date ***
PC: @          0x32acb84 starrocks::pipeline::QueryContextManager::collect_query_statistics()
*** SIGSEGV (@0x38) received by PID 30662 (TID 0x7f1b00698700) from PID 56; stack trace: ***
    @          0x6641b82 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f1f016c394b os::Linux::chained_handler()
    @     0x7f1f016c8b6d JVM_handle_linux_signal
    @     0x7f1f016bc1b8 signalHandler()
    @     0x7f1f00855630 (unknown)
    @          0x32acb84 starrocks::pipeline::QueryContextManager::collect_query_statistics()
    @          0x5da60d4 starrocks::PInternalServiceImplBase<>::collect_query_statistics()
    @          0x6874ccd brpc::policy::ProcessRpcRequest()
    @          0x6955797 brpc::ProcessInputMessage()
    @          0x695666b brpc::InputMessenger::OnNewMessages()
    @          0x67fb74e brpc::Socket::ProcessEvent()
    @          0x67a061f bthread::TaskGroup::task_runner()
    @          0x68e3e91 bthread_make_fcontext

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/34316
Jira
问题版本:
- 3.0.0 ～ 3.0.7
- 3.1.0 ～ 3.1.4
修复版本:
- 3.0.8+
- 3.1.5+
问题原因:
临时解决办法:
- 无

trueeyu · 2024年03月5日 04:00

Global Runtime Filter crash

*** Aborted at 1700708505 (unix time) try "date -d @1700708505" if you are using GNU date ***
PC: @     0x7fd3a38b36a6 __memcpy_ssse3_back
*** SIGSEGV (@0x1ffff0) received by PID 3045 (TID 0x7fd329f54700) from PID 2097136; stack trace: ***
    @          0x63a11c2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fd3a52a1cfb os::Linux::chained_handler()
    @     0x7fd3a52a686e JVM_handle_linux_signal
    @     0x7fd3a529aad8 signalHandler()
    @     0x7fd3a4462630 (unknown)
    @     0x7fd3a38b36a6 __memcpy_ssse3_back
    @          0x43ecae9 starrocks::JoinRuntimeFilter::serialize()
    @          0x2c41066 starrocks::RuntimeBloomFilter<>::serialize()
    @          0x43cf16d starrocks::RuntimeFilterHelper::serialize_runtime_filter()
    @          0x4e32e1b starrocks::RuntimeFilterPort::publish_runtime_filters()
    @          0x2e5b879 starrocks::pipeline::HashJoinBuildOperator::set_finishing()
    @          0x2ac99e7 starrocks::pipeline::PipelineDriver::_mark_operator_finishing()
    @          0x2acb8fc starrocks::pipeline::PipelineDriver::process()
    @          0x56a4e5e starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x4f8afa2 starrocks::ThreadPool::dispatch_thread()
    @          0x4f85a3a starrocks::Thread::supervise_thread()
    @     0x7fd3a445aea5 start_thread
    @     0x7fd3a385bb0d __clone
    @                0x0 (unknown)

或

*** Aborted at 1688607906 (unix time) try "date -d @1688607906" if you are using GNU date ***
PC: @     0x7fcb2a79aa19 __memmove_avx_unaligned_erms
*** SIGSEGV (@0x7f8a86400000) received by PID 10757 (TID 0x7fca8dba8700) from PID 18446744071666925568; stack trace: ***
    @          0x5824342 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fcb2b19fc20 (unknown)
    @     0x7fcb2a79aa19 __memmove_avx_unaligned_erms
    @          0x3e953d6 starrocks::vectorized::JoinRuntimeFilter::serialize()
    @          0x3e6a213 starrocks::vectorized::RuntimeBloomFilter<>::serialize()
    @          0x3e60b12 starrocks::vectorized::RuntimeFilterHelper::serialize_runtime_filter()
    @          0x47a1a20 starrocks::RuntimeFilterPort::publish_runtime_filters()
    @          0x2fd8afb starrocks::pipeline::HashJoinBuildOperator::set_finishing()
    @          0x2ca6bb9 starrocks::pipeline::PipelineDriver::_mark_operator_finishing()
    @          0x2ca6c69 starrocks::pipeline::PipelineDriver::_mark_operator_finished()
    @          0x2ca722b starrocks::pipeline::PipelineDriver::_mark_operator_cancelled()
    @          0x2ca780a starrocks::pipeline::PipelineDriver::cancel_operators()
    @          0x4ec3e4f starrocks::pipeline::PipelineDriverPoller::run_internal()
    @          0x48be68a starrocks::Thread::supervise_thread()
    @     0x7fcb2b19517a start_thread
    @     0x7fcb2a736df3 __GI___clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/32219
- https://github.com/StarRocks/starrocks/pull/35776
Jira
问题版本:
- 2.5.0 ~ 2.5.16
- 3.0.0~3.0.8
- 3.1.0~3.1.5
修复版本:
- 2.5.17+
- 3.0.9+
- 3.1.6+
问题原因:
临时解决办法:
- set global enable_global_runtime_filter = false

trueeyu · 2024年05月20日 11:46

remove_expired_versions crash

*** Aborted at 1699314386 (unix time) try "date -d @1699314386" if you are using GNU date ***
PC: @          0x8046b03 std::_Rb_tree_rebalance_for_erase()
*** SIGSEGV (@0x0) received by PID 1950245 (TID 0x7fc7ae8fd700) from PID 0; stack trace: ***
    @          0x5ba1a42 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fc9499b07fb os::Linux::chained_handler()
    @     0x7fc9499b54bd JVM_handle_linux_signal
    @     0x7fc9499a7e78 signalHandler()
    @     0x7fc9486a9cf0 (unknown)
    @          0x8046b03 std::_Rb_tree_rebalance_for_erase()
    @          0x42db369 starrocks::TabletUpdates::remove_expired_versions()
    @          0x4294fa0 starrocks::TabletManager::start_trash_sweep()
    @          0x424aefb starrocks::StorageEngine::_start_trash_sweep()
    @          0x44c3620 starrocks::StorageEngine::_garbage_sweeper_thread_callback()
    @          0x80b4f20 execute_native_thread_routine
    @     0x7fc94869f1ca start_thread
    @     0x7fc94830be73 __GI___clone
    @                0x0 (unknown)

或

*** Aborted at 1693450540 (unix time) try "date -d @1693450540" if you are using GNU date ***

PC: @ 0x459fa2d starrocks::ParsedPageV2::read()

*** SIGSEGV (@0xfffffffffffffffc) received by PID 1205872 (TID 0x7feaec60f700) from PID 18446744073709551612; stack trace: ***

 @ 0x5960be2 google::(anonymous namespace)::FailureSignalHandler()

 @ 0x7fec5d00ec20 (unknown)

 @ 0x459fa2d starrocks::ParsedPageV2::read()

 @ 0x4573fcb starrocks::ScalarColumnIterator::next_batch()

 @ 0x4164573 starrocks::vectorized::SegmentIterator::_read()

 @ 0x415ad8c starrocks::vectorized::SegmentIterator::_do_get_next()

 @ 0x415e380 starrocks::vectorized::SegmentIterator::do_get_next()

 @ 0x41e2ab2 starrocks::vectorized::ProjectionIterator::do_get_next()

 @ 0x4786ee5 starrocks::SegmentIteratorWrapper::do_get_next()

 @ 0x45b6723 starrocks::vectorized::TimedChunkIterator::do_get_next()

 @ 0x420b30e starrocks::vectorized::TabletReader::do_get_next()

 @ 0x2fd2dfb starrocks::pipeline::OlapChunkSource::_read_chunk_from_storage()

 @ 0x2fd34db starrocks::pipeline::OlapChunkSource::_read_chunk()

 @ 0x2fc2dcf starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking()

 @ 0x2d410b4 _ZZN9starrocks8pipeline12ScanOperator18_trigger_next_scanEPNS_12RuntimeStateEiENKUlvE_clEv

 @ 0x2d521ae starrocks::workgroup::ScanExecutor::worker_thread()

 @ 0x4983a52 starrocks::ThreadPool::dispatch_thread()

 @ 0x497e54a starrocks::Thread::supervise_thread()

 @ 0x7fec5d00417a start_thread

 @ 0x7fec5c5a5dc3 __GI___clone

 @ 0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/34814
Jira
问题版本:
- 2.5.0 ~ 2.5.14
- 3.0.0 ~ 3.0.8
- 3.1.0 ~ 3.1.4
修复版本:
- 2.5.15+
- 3.0.9+
- 3.1.5+
问题原因:
- 没有加锁
临时解决办法:

trueeyu · 2023年12月13日 02:42

Spill crash

*** Aborted at 1701857859 (unix time) try "date -d @1701857859" if you are using GNU date ***
PC: @          0x2e10aa2 _ZNSt17_Function_handlerIFN9starrocks6StatusEvEZZNS0_8pipeline34SpillablePartitionSortSinkOperator13set_finishingEPNS0_12RuntimeStateEENKUlS6_T_E0_clISt10shared_ptrINS0_5spill14IOTaskExecutorEEEEDaS6_S7_EUlvE_E9_M_invokeERKSt9_An
y_data
*** SIGSEGV (@0x0) received by PID 2438961 (TID 0x7f37bb86d700) from PID 0; stack trace: ***
    @          0x6387fa2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f39722bacf0 (unknown)
    @          0x2e10aa2 _ZNSt17_Function_handlerIFN9starrocks6StatusEvEZZNS0_8pipeline34SpillablePartitionSortSinkOperator13set_finishingEPNS0_12RuntimeStateEENKUlS6_T_E0_clISt10shared_ptrINS0_5spill14IOTaskExecutorEEEEDaS6_S7_EUlvE_E9_M_invokeERKSt9_An
y_data
    @          0x2dde13d _ZNSt17_Function_handlerIFN9starrocks6StatusEvEZNS0_5spill7Spiller23set_flush_all_call_backINS3_23ResourceMemTrackerGuardIJSt8weak_ptrINS0_8pipeline12QueryContextEES7_IS4_EEEEEES1_RKSt8functionIS2_EPNS0_12RuntimeStateERNS3_14IOTa
skExecutorERKT_EUlvE_E9_M_invokeERKSt9_Any_data
    @          0x2e773a5 starrocks::spill::SpillerWriter::_decrease_running_flush_tasks()
    @          0x2dd1f27 _ZZZN9starrocks5spill16RawSpillerWriter5flushIRNS0_14IOTaskExecutorERNS0_23ResourceMemTrackerGuardIJSt8weak_ptrINS_8pipeline12QueryContextEES6_INS0_7SpillerEEEEEEENS_6StatusEPNS_12RuntimeStateEOT_OT0_ENKUlvE0_clEvENKUlvE1_clEv
    @          0x2dd5ad7 _ZNSt17_Function_handlerIFvvEZN9starrocks5spill16RawSpillerWriter5flushIRNS2_14IOTaskExecutorERNS2_23ResourceMemTrackerGuardIJSt8weak_ptrINS1_8pipeline12QueryContextEES8_INS2_7SpillerEEEEEEENS1_6StatusEPNS1_12RuntimeStateEOT_OT0_
EUlvE0_E9_M_invokeERKSt9_Any_data
    @          0x2b017b1 starrocks::workgroup::ScanExecutor::worker_thread()
    @          0x4f782b2 starrocks::ThreadPool::dispatch_thread()
    @          0x4f72d4a starrocks::Thread::supervise_thread()
    @     0x7f39722b01ca start_thread
    @     0x7f3971576e73 __GI___clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/36223
- https://github.com/StarRocks/starrocks/pull/36351
Jira
问题版本:
- 3.0.0 ~ 3.0.8
- 3.1.0 ~ 3.1.5
修复版本:
- 3.0.9+
- 3.1.6+
问题原因:
临时解决办法:
- 关闭 spill: set global enable_spill=false;

trueeyu · 2023年12月13日 06:27

存算分离模式下，get_tablet_stats crash

*** Aborted at 1692934579 (unix time) try "date -d @1692934579" if you are using GNU date ***
PC: @          0x95af4bd bthread_mutex_unlock
*** SIGSEGV (@0x1) received by PID 2750248 (TID 0x7fb460dff640) from PID 1; stack trace: ***
    @          0x943287a google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fb4e1842520 (unknown)
    @          0x95af4bd bthread_mutex_unlock
    @          0x848a49a _ZNSt17_Function_handlerIFvvEZN9starrocks15LakeServiceImpl16get_tablet_statsEPN6google8protobuf13RpcControllerEPKNS1_4lake17TabletStatRequestEPNS7_18TabletStatResponseEPNS4_7ClosureEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x85b429b starrocks::ThreadPool::dispatch_thread()
    @          0x85ae62a starrocks::Thread::supervise_thread()
    @     0x7fb4e1894b43 (unknown)
    @     0x7fb4e1926a00 (unknown)
    @                0x0 (unknown)

Github Issue:
- https://github.com/StarRocks/starrocks/issues/29929
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/29930
Jira
问题版本:
- 3.0.0 ~ 3.0.5
- 3.1.0 ~ 3.1.2
修复版本:
- 3.0.6+
- 3.1.3+
问题原因:
临时解决办法:

trueeyu · 2023年12月18日 11:01

Spill crash

*** Aborted at 1702867669 (unix time) try "date -d @1702867669" if you are using GNU date ***
PC: @          0x32d6d38 starrocks::MergeCursorsCascade::init()
*** SIGSEGV (@0x0) received by PID 15893 (TID 0x7fdc0f5bf700) from PID 0; stack trace: ***
    @          0x6240182 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fde620c8630 (unknown)
    @          0x32d6d38 starrocks::MergeCursorsCascade::init()
    @          0x4f127db starrocks::CascadeChunkMerger::init()
    @          0x346548d starrocks::spill::OrderedInputStream::init()
    @          0x3465c1e starrocks::spill::BlockGroup::as_ordered_stream()
    @          0x34567f8 starrocks::spill::RawSpillerWriter::acquire_stream()
    @          0x345d2d5 starrocks::spill::Spiller::_acquire_input_stream()
    @          0x33fa3cd _ZNSt17_Function_handlerIFN9starrocks6StatusEvEZNS0_5spill7Spiller23set_flush_all_call_backERKSt8functionIS2_EPNS0_12RuntimeStateERNS3_14IOTaskExecutorERKNS3_15MemTrackerGuardEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x3456395 starrocks::spill::SpillerWriter::_decrease_running_flush_tasks()
    @          0x345b546 starrocks::spill::RawSpillerWriter::set_flush_all_call_back()
    @          0x340e31d _ZZN9starrocks8pipeline38SpillableAggregateBlockingSinkOperator13set_finishingEPNS_12RuntimeStateEENKUlS3_T_E0_clISt10shared_ptrINS_5spill14IOTaskExecutorEEEEDaS3_S4_
    @          0x340f162 starrocks::pipeline::SpillableAggregateBlockingSinkOperator::set_finishing()
    @          0x30c6817 starrocks::pipeline::PipelineDriver::_mark_operator_finishing()
    @          0x30c6929 starrocks::pipeline::PipelineDriver::_mark_operator_finished()
    @          0x30c6fdb starrocks::pipeline::PipelineDriver::_mark_operator_cancelled()
    @          0x30c769a starrocks::pipeline::PipelineDriver::cancel_operators()
    @          0x56fc80f starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x506b022 starrocks::ThreadPool::dispatch_thread()
    @          0x5065b1a starrocks::Thread::supervise_thread()
    @     0x7fde620c0ea5 start_thread
    @     0x7fde616dbb0d __clone
    @                0x0 (unknown)

或

*** Aborted at 1688627331 (unix time) try "date -d @1688627331" if you are using GNU date ***
PC: @     0x7f7b534e69d5 __GI_raise
*** SIGABRT (@0x3e900237c64) received by PID 2325604 (TID 0x7f7a69fef640) from PID 2325604; stack trace: ***
    @          0x59bd9c2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f7b5369b1d0 (unknown)
    @     0x7f7b534e69d5 __GI_raise
    @     0x7f7b534cf894 __GI_abort
    @          0x237c0d2 _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold
    @          0x787cad6 __cxxabiv1::__terminate()
    @          0x787cb41 std::terminate()
    @          0x787d1ff __cxa_pure_virtual
    @          0x2997fab starrocks::spill::RawSpillerWriter::acquire_stream()
    @          0x28f7603 _ZZN9starrocks5spill24PartitionedSpillerWriter5flushIRNS0_14IOTaskExecutorERNS0_23ResourceMemTrackerGuardIJSt8weak_ptrINS_8pipeline12QueryContextEEEEEEENS_6StatusEPNS_12RuntimeStateEOT_OT0_ENKUlvE0_clEv
    @          0x28f8c84 _ZNSt17_Function_handlerIFvvEZN9starrocks5spill24PartitionedSpillerWriter5flushIRNS2_14IOTaskExecutorERNS2_23ResourceMemTrackerGuardIJSt8weak_ptrINS1_8pipeline12QueryContextEEEEEEENS1_6StatusEPNS1_12RuntimeStateEOT_OT0_EUlvE0_E9_M_invokeERKSt9_Any_data
    @          0x2624271 starrocks::workgroup::ScanExecutor::worker_thread()
    @          0x488f0f2 starrocks::ThreadPool::dispatch_thread()
    @          0x4889bea starrocks::Thread::supervise_thread()
    @     0x7f7b536903fb start_thread
    @     0x7f7b535acc23 __GI___clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/26738
Jira
问题版本:
- 3.0.0 ~ 3.0.3
修复版本:
- 3.0.4+
问题原因:
临时解决办法:
- 关闭 spill: set global enable_spill=false;

trueeyu · 2023年12月19日 04:05

Agg Crash

*** Aborted at 1702555022 (unix time) try "date -d @1702555022" if you are using GNU date ***
PC: @          0x26ca130 starrocks::vectorized::TDistinctAggregateFunction<>::merge()
*** SIGSEGV (@0x2ad195326000) received by PID 1513 (TID 0x2ad102607700) from PID 18446744071917690880; stack trace: ***
    @          0x4875742 google::(anonymous namespace)::FailureSignalHandler()
    @     0x2ad0cd8e2630 (unknown)
    @          0x26ca130 starrocks::vectorized::TDistinctAggregateFunction<>::merge()
    @          0x2501d23 starrocks::vectorized::NullableAggregateFunctionBase<>::merge_batch()
    @          0x228c98d starrocks::Aggregator::compute_batch_agg_states()
    @          0x2232738 starrocks::pipeline::AggregateBlockingSinkOperator::push_chunk()
    @          0x1e5c71c starrocks::pipeline::PipelineDriver::process()
    @          0x3ce10ad starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x37a7915 starrocks::ThreadPool::dispatch_thread()
    @          0x37a2e3a starrocks::Thread::supervise_thread()
    @     0x2ad0cd8daea5 start_thread
    @     0x2ad0ce2fb96d __clone
    @                0x0 (unknown)

Github Issue:
- https://github.com/StarRocks/starrocks/issues/19901
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/19906
Jira
问题版本:
- 2.3.0 ~ 2.3.10
- 2.4.0 ~ 2.4.4
- 2.5.0 ~ 2.5.3
修复版本:
- 2.3.11+
- 2.4.5+
- 2.5.4+
问题原因:
临时解决办法:
- set global cbo_enable_low_cardinality_optimize=false;

trueeyu · 2023年12月19日 06:12

HashJoin crash

*** Aborted at 1702536561 (unix time) try "date -d @1702536561" if you are using GNU date ***
PC: @          0x2d44521 starrocks::vectorized::FixedLengthColumnBase<>::serialize()
*** SIGSEGV (@0x2c126d41b000) received by PID 23691 (TID 0x2b7c47dea700) from PID 1833021440; stack trace: ***
    @          0x5bfc642 google::(anonymous namespace)::FailureSignalHandler()
    @     0x2b7a336657fb os::Linux::chained_handler()
    @     0x2b7a3366a4bd JVM_handle_linux_signal
    @     0x2b7a3365ce78 signalHandler()
    @     0x2b7a33d59630 (unknown)
    @          0x2d44521 starrocks::vectorized::FixedLengthColumnBase<>::serialize()
    @          0x2e5a7e8 starrocks::vectorized::SerializedJoinBuildFunc::_build_columns()
    @          0x2e5dbf2 starrocks::vectorized::SerializedJoinBuildFunc::construct_hash_table()
    @          0x2e5f381 starrocks::vectorized::JoinHashTable::build()
    @          0x322a187 starrocks::vectorized::HashJoiner::_build()
    @          0x322a30c starrocks::vectorized::HashJoiner::build_ht()
    @          0x30dbc8c starrocks::pipeline::HashJoinBuildOperator::set_finishing()
    @          0x2da6e59 starrocks::pipeline::PipelineDriver::_mark_operator_finishing()
    @          0x2da6f09 starrocks::pipeline::PipelineDriver::_mark_operator_finished()
    @          0x2da74fb starrocks::pipeline::PipelineDriver::_mark_operator_cancelled()
    @          0x2da7aea starrocks::pipeline::PipelineDriver::cancel_operators()
    @          0x51d3ca7 starrocks::pipeline::PipelineDriverPoller::run_internal()
    @          0x4bb586a starrocks::Thread::supervise_thread()
    @     0x2b7a33d51ea5 start_thread
    @     0x2b7a34772b0d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/32219
Jira
问题版本:
- 2.3.0 ~ 2.3.18
- 2.4.0 ~ latest
- 2.5.0 ~ 2.5.16
- 3.0.0 ~ 3.0.6
- 3.1.0 ~ 3.1.3
修复版本:
- 2.3.19+
- 2.4 未修复
- 2.5.17+
- 3.0.7+
- 3.1.4+
问题原因:
临时解决办法:

trueeyu · 2023年12月28日 07:39

存算分离版本 _column_index_mem_usage crash

*** Aborted at 1703239184 (unix time) try "date -d @1703239184" if you are using GNU date ***
PC: @          0x4aab274 starrocks::ColumnReader::mem_usage()
*** SIGSEGV (@0x0) received by PID 13993 (TID 0x7f570561f700) from PID 0; stack trace: ***
    @          0x6387fa2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f5793e6c630 (unknown)
    @          0x4aab274 starrocks::ColumnReader::mem_usage()
    @          0x463d07d starrocks::Segment::_column_index_mem_usage()
    @          0x47a9226 starrocks::lake::TabletManager::cache_segment()
    @          0x47a92f3 starrocks::lake::TabletManager::update_segment_cache_size()
    @          0x4641611 starrocks::Segment::open()
    @          0x47a0f18 starrocks::lake::Tablet::load_segment()
    @          0x4792f76 starrocks::lake::Rowset::load_segments()
    @          0x4794fee starrocks::lake::Rowset::read()
    @          0x47b49b5 starrocks::lake::TabletReader::get_segment_iterators()
    @          0x47b5a41 starrocks::lake::TabletReader::init_collector()
    @          0x47b720b starrocks::lake::TabletReader::open()
    @          0x55c6e84 starrocks::connector::LakeDataSource::init_tablet_reader()
    @          0x55c7844 starrocks::connector::LakeDataSource::open()
    @          0x2afb401 starrocks::pipeline::ConnectorChunkSource::_open_data_source()
    @          0x2afc4e1 starrocks::pipeline::ConnectorChunkSource::_read_chunk()
    @          0x2db0867 starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking()
    @          0x2af8d0b _ZZN9starrocks8pipeline12ScanOperator18_trigger_next_scanEPNS_12RuntimeStateEiENKUlvE_clEv
    @          0x2b017b1 starrocks::workgroup::ScanExecutor::worker_thread()
    @          0x4f782b2 starrocks::ThreadPool::dispatch_thread()
    @          0x4f72d4a starrocks::Thread::supervise_thread()
    @     0x7f5793e64ea5 start_thread
    @     0x7f5793265b0d __clone
    @                0x0 (unknown)

Github Issue:
- https://github.com/StarRocks/starrocks/issues/34273
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/34283
Jira
问题版本:
- 3.1.0 ~ 3.1.4
修复版本:
- 3.1.5+
问题原因:
临时解决办法:

hz_yanglei · 2024年01月3日 02:47

有pr吗

trueeyu · 2024年01月22日 03:15

array_overlap crash

*** Aborted at 1705652227 (unix time) try "date -d @1705652227" if you are using GNU date ***
PC: @          0x4dbc078 starrocks::vectorized::ArrayColumn::get()
*** SIGSEGV (@0x1010) received by PID 10923 (TID 0x7f0b95ff0700) from PID 4112; stack trace: ***
    @          0x5824342 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f0e366013c2 os::Linux::chained_handler()
    @     0x7f0e36608196 JVM_handle_linux_signal
    @     0x7f0e365fe253 signalHandler()
    @     0x7f0e35ac9630 (unknown)
    @          0x4dbc078 starrocks::vectorized::ArrayColumn::get()
    @          0x4d2e38e starrocks::vectorized::ArrayOverlap<>::_array_overlap_item<>()
    @          0x4d2ecd9 starrocks::vectorized::ArrayOverlap<>::_array_overlap<>()
    @          0x4d2f313 starrocks::vectorized::ArrayFunctions::array_overlap_varchar()
    @          0x3ddde57 starrocks::vectorized::VectorizedFunctionCallExpr::evaluate()
    @          0x37c586e starrocks::ExprContext::evaluate()
    @          0x2cde821 starrocks::eager_prune_eval_conjuncts()
    @          0x2ce05d6 starrocks::ExecNode::eval_conjuncts()
    @          0x2f5c505 starrocks::pipeline::OlapChunkSource::_read_chunk_from_storage()
    @          0x2f5c96b starrocks::pipeline::OlapChunkSource::_read_chunk()
    @          0x2f4c34c starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking()
    @          0x2ccaca4 _ZZN9starrocks8pipeline12ScanOperator18_trigger_next_scanEPNS_12RuntimeStateEiENKUlvE_clEv
    @          0x2cdbd5e starrocks::workgroup::ScanExecutor::worker_thread()
    @          0x48c3b92 starrocks::ThreadPool::dispatch_thread()
    @          0x48be68a starrocks::Thread::supervise_thread()
    @     0x7f0e35ac1ea5 start_thread
    @     0x7f0e350dcb0d __clone
    @                0x0 (unknown)

Github Issue:
- https://github.com/StarRocks/starrocks/issues/24587
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/24586
Jira
问题版本:
- 2.3.0~2.3.13
- 2.4.0~2.4.5
- 2.5.0~2.5.6
- 3.0.0~3.0.1
修复版本:
- 2.3.14+
- 2.4.6+
- 2.5.7+
- 3.0.2+
问题原因:
临时解决办法:

trueeyu · 2024年01月26日 02:05

Export / Select into outfile / insert into files crash

*** Aborted at 1703119356 (unix time) try "date -d @1703119356" if you are using GNU date ***
PC: @          0x6581ac7 je_large_dalloc
*** SIGSEGV (@0x0) received by PID 44953 (TID 0x7ff180a2a700) from PID 0; stack trace: ***
    @          0x5b1ba42 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7ff28c0aee92 os::Linux::chained_handler()
    @     0x7ff28c0b5526 JVM_handle_linux_signal
    @     0x7ff28c0abb03 signalHandler()
    @     0x7ff28b57e5d0 (unknown)
    @          0x6581ac7 je_large_dalloc
    @          0x51285f8 std::_Rb_tree<>::_M_erase()
    @          0x51285b9 std::_Rb_tree<>::_M_erase()
    @          0x5129f21 starrocks::pipeline::ExportSinkIOBuffer::_open_file_writer()
    @          0x512a48f starrocks::pipeline::ExportSinkIOBuffer::_process_chunk()
    @          0x51282db starrocks::pipeline::SinkIOBuffer::execute_io_task()
    @          0x5c6252c bthread::ExecutionQueueBase::_execute()
    @          0x5c632a8 bthread::ExecutionQueueBase::_execute_tasks()
    @          0x494dff0 starrocks::PriorityThreadPool::work_thread()
    @          0x5adb407 thread_proxy
    @     0x7ff28b576dd5 start_thread
    @     0x7ff28ab9202d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
Jira
问题版本:
- 2.5 ~ latest
- 3.0 ~ latest
- 3.1 ~ latest
- 3.2 ~ latest
修复版本:
- 2.5 未修复
- 3.0 未修复
- 3.1 未修复
- 3.2 未修复
缓解版本:
- 2.5.19+
- 3.0.10+
- 3.1.0+
问题原因:
临时解决办法:

trueeyu · 2024年01月26日 12:15

大量删除 Tablet 后导致 BE 建表慢

有如下堆栈:

tid: 4925
         0x5ae583f  rocksdb::MemTableIterator::key()
         0x5afb41e  rocksdb::FragmentedRangeTombstoneList::FragmentTombstones()
         0x5afbc8f  rocksdb::FragmentedRangeTombstoneList::FragmentedRangeTombstoneList()
         0x5ae0376  rocksdb::MemTable::NewRangeTombstoneIterator()
         0x5ae0654  rocksdb::MemTable::Get()
         0x5a22f4f  rocksdb::DBImpl::GetImpl()
         0x5a23c41  rocksdb::DBImpl::Get()
         0x45731cd  starrocks::KVStore::get()
         0x42e7655  starrocks::TabletMetaManager::get_persistent_index_meta()
         0x433067b  starrocks::TabletUpdates::get_pk_index_write_amp_score()
         0x42cc631  starrocks::TabletManager::pick_tablets_to_do_pk_index_major_compaction()
         0x4555c37  starrocks::PersistentIndexCompactionManager::schedule()
         0x457c630  _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN9starrocks13StorageEngine16start_bg_threadsEvEUlvE5_EEEEE6_M_runEv
         0x7eda630  execute_native_thread_routine
    0x7f96359c5ea5  start_thread
    0x7f96356ee8dd  __clone
             (nil)  (unknown)
             
2 tids: 4844,4846
    0x7f96359c939c  __GI___pthread_rwlock_wrlock
         0x42d63df  starrocks::TabletManager::create_tablet()
         0x42838af  starrocks::StorageEngine::create_tablet()
         0x2747291  starrocks::run_create_tablet_task()
         0x4cc3e32  starrocks::ThreadPool::dispatch_thread()
         0x4cbe8ca  starrocks::Thread::supervise_thread()
    0x7f96359c5ea5  start_thread
    0x7f96356ee8dd  __clone
             (nil)  (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/38299
Jira
问题版本:
- 2.5.0 ~ 2.5.17
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.6
- 3.2.1 ~ 3.2.2
修复版本:
- 2.5.18+
- 3.0.10+
- 3.1.7+
- 3.2.3+
问题原因:
临时解决办法:

trueeyu · 2024年01月29日 06:07

同步物化视图 insert into crash

一般是物化视图的列中有大写字母会导致

*** Aborted at 1706016682 (unix time) try "date -d @1706016682" if you are using GNU date ***
PC: @          0x3a75a40 starrocks::BinaryColumnBase<>::append_selective()
*** SIGSEGV (@0x4) received by PID 31 (TID 0x7fe9dbffd640) from PID 4; stack trace: ***
    @          0x79bc8ea google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fea1a7d9896 os::Linux::chained_handler()
    @     0x7fea1a7df48b JVM_handle_linux_signal
    @     0x7fea1a7d20fc signalHandler()
    @     0x7fea1979b520 (unknown)
    @          0x3a75a40 starrocks::BinaryColumnBase<>::append_selective()
    @          0x60fac89 starrocks::MemTable::insert()
    @          0x60f03be starrocks::DeltaWriter::write()
    @          0x63cfebe starrocks::AsyncDeltaWriter::_execute()
    @          0x7b3a41c bthread::ExecutionQueueBase::_execute()
    @          0x7b3b298 bthread::ExecutionQueueBase::_execute_tasks()
    @          0x66202eb starrocks::ThreadPool::dispatch_thread()
    @          0x661a58a starrocks::Thread::supervise_thread()
    @     0x7fea197edb43 (unknown)
    @     0x7fea1987fa00 (unknown)
    @                0x0 (unknown)

Github Issue:
- https://github.com/StarRocks/starrocks/issues/31504
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/31505
Jira
问题版本:
- 3.1.0 ～ 3.1.3
修复版本:
- 3.1.4+
问题原因:
解决办法:
- 直接升级不能解决问题。需要删除有问题的同步物化视图，并升级到已修复版本。

trueeyu · 2024年01月29日 09:49

Union 报错: This sql plan has multi result sinks

一般有 CN 结点时出现

2024-01-26 11:04:20,527 WARN (starrocks-mysql-nio-pool-33031|282502) [StmtExecutor.execute():552] execute Exception, sql select 's'^M
union all^M
select org_no from dim.dim_org
com.starrocks.sql.common.StarRocksPlannerException: This sql plan has multi result sinks
        at com.starrocks.qe.Coordinator.computeFragmentExecParams(Coordinator.java:1702) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.Coordinator.prepareExec(Coordinator.java:640) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.Coordinator.exec(Coordinator.java:685) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:768) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:466) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:326) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:443) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:710) ~[starrocks-fe.jar:?]
        at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/25832
Jira
问题版本:
- 2.5.0 ～ 2.5.18
- 3.0.0 ～ 3.0.9
修复版本:
- 2.5.19+
- 3.0.10+
问题原因:
临时解决办法:
- set prefer_compute_node = false;

trueeyu · 2024年01月30日 03:27

DataStreamRecvr profile use-after-free

*** Aborted at 1700492373 (unix time) try "date -d @1700492373" if you are using GNU date ***
PC: @          0x32e5e30 starrocks::ScopedTimer<>::~ScopedTimer()
*** SIGSEGV (@0x0) received by PID 13416 (TID 0x7f4fa6d36700) from PID 0; stack trace: ***
    @          0x66b6ec2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f50274d0cfb os::Linux::chained_handler()
    @     0x7f50274d59bd JVM_handle_linux_signal
    @     0x7f50274c9ad8 signalHandler()
    @     0x7f5026691630 (unknown)
    @          0x32e5e30 starrocks::ScopedTimer<>::~ScopedTimer()
    @          0x5368545 starrocks::DataStreamRecvr::PipelineSenderQueue::try_to_build_chunk_meta()
    @          0x5372bbb starrocks::DataStreamRecvr::PipelineSenderQueue::add_chunks<>()
    @          0x536ab52 starrocks::DataStreamRecvr::PipelineSenderQueue::add_chunks()
    @          0x52ddbeb starrocks::DataStreamRecvr::add_chunks()
    @          0x527d63f starrocks::DataStreamMgr::transmit_chunk()
    @          0x5e107cc starrocks::PInternalServiceImplBase<>::_transmit_chunk()
    @          0x52a3600 starrocks::PriorityThreadPool::work_thread()
    @          0x6676887 thread_proxy
    @     0x7f5026689ea5 start_thread
    @     0x7f5025ca4b0d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/36780
Jira
问题版本:
- 2.5.0 ~ 2.5.16
- 3.0.0 ~ 3.0.8
- 3.1.0 ~ 3.1.5
- 3.2.0
修复版本:
- 2.5.17+
- 3.0.9+
- 3.1.6+
问题原因:
临时解决办法:

常见 Crash / BUG / 优化 查询

Join reorder + 窗口函数导致 crash

smp_call_function_many 占用大量CPU

collect_query_statistics crash

Global Runtime Filter crash

remove_expired_versions crash

Spill crash

存算分离模式下，get_tablet_stats crash

Spill crash

Agg Crash

HashJoin crash

存算分离版本 _column_index_mem_usage crash

array_overlap crash

Export / Select into outfile / insert into files crash

大量删除 Tablet 后导致 BE 建表慢

同步物化视图 insert into crash

Union 报错: This sql plan has multi result sinks

DataStreamRecvr profile use-after-free

常见 Crash / BUG / 优化查询