常见 Crash / BUG / 优化 查询

💬 StarRocks 用户问答
#184

  1. 同步物化视图 insert into crash

一般是物化视图的列中有大写字母会导致

*** Aborted at 1706016682 (unix time) try "date -d @1706016682" if you are using GNU date ***
PC: @          0x3a75a40 starrocks::BinaryColumnBase<>::append_selective()
*** SIGSEGV (@0x4) received by PID 31 (TID 0x7fe9dbffd640) from PID 4; stack trace: ***
    @          0x79bc8ea google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fea1a7d9896 os::Linux::chained_handler()
    @     0x7fea1a7df48b JVM_handle_linux_signal
    @     0x7fea1a7d20fc signalHandler()
    @     0x7fea1979b520 (unknown)
    @          0x3a75a40 starrocks::BinaryColumnBase<>::append_selective()
    @          0x60fac89 starrocks::MemTable::insert()
    @          0x60f03be starrocks::DeltaWriter::write()
    @          0x63cfebe starrocks::AsyncDeltaWriter::_execute()
    @          0x7b3a41c bthread::ExecutionQueueBase::_execute()
    @          0x7b3b298 bthread::ExecutionQueueBase::_execute_tasks()
    @          0x66202eb starrocks::ThreadPool::dispatch_thread()
    @          0x661a58a starrocks::Thread::supervise_thread()
    @     0x7fea197edb43 (unknown)
    @     0x7fea1987fa00 (unknown)
    @                0x0 (unknown)
#185

  1. Union 报错: This sql plan has multi result sinks

一般有 CN 结点时出现

2024-01-26 11:04:20,527 WARN (starrocks-mysql-nio-pool-33031|282502) [StmtExecutor.execute():552] execute Exception, sql select 's'^M
union all^M
select org_no from dim.dim_org
com.starrocks.sql.common.StarRocksPlannerException: This sql plan has multi result sinks
        at com.starrocks.qe.Coordinator.computeFragmentExecParams(Coordinator.java:1702) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.Coordinator.prepareExec(Coordinator.java:640) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.Coordinator.exec(Coordinator.java:685) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:768) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:466) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:326) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:443) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:710) ~[starrocks-fe.jar:?]
        at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]
#186

  1. DataStreamRecvr profile use-after-free

*** Aborted at 1700492373 (unix time) try "date -d @1700492373" if you are using GNU date ***
PC: @          0x32e5e30 starrocks::ScopedTimer<>::~ScopedTimer()
*** SIGSEGV (@0x0) received by PID 13416 (TID 0x7f4fa6d36700) from PID 0; stack trace: ***
    @          0x66b6ec2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f50274d0cfb os::Linux::chained_handler()
    @     0x7f50274d59bd JVM_handle_linux_signal
    @     0x7f50274c9ad8 signalHandler()
    @     0x7f5026691630 (unknown)
    @          0x32e5e30 starrocks::ScopedTimer<>::~ScopedTimer()
    @          0x5368545 starrocks::DataStreamRecvr::PipelineSenderQueue::try_to_build_chunk_meta()
    @          0x5372bbb starrocks::DataStreamRecvr::PipelineSenderQueue::add_chunks<>()
    @          0x536ab52 starrocks::DataStreamRecvr::PipelineSenderQueue::add_chunks()
    @          0x52ddbeb starrocks::DataStreamRecvr::add_chunks()
    @          0x527d63f starrocks::DataStreamMgr::transmit_chunk()
    @          0x5e107cc starrocks::PInternalServiceImplBase<>::_transmit_chunk()
    @          0x52a3600 starrocks::PriorityThreadPool::work_thread()
    @          0x6676887 thread_proxy
    @     0x7f5026689ea5 start_thread
    @     0x7f5025ca4b0d __clone
    @                0x0 (unknown)
#187

  1. Apply 线程数不够导致 crash

F1122 01:12:48.527740 150889 tablet_updates.cpp:788] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:432759625 #version:88 [30 98@86 98.1] pending: rowsets:1

*** Aborted at 1700375600 (unix time) try "date -d @1700375600" if you are using GNU date ***
PC: @     0x2b9e602f21f7 __GI_raise
*** SIGABRT (@0x7d00001111f) received by PID 69919 (TID 0x2ba20fb33700) from PID 69919; stack trace: ***
    @          0x63c5622 google::(anonymous namespace)::FailureSignalHandler()
    @     0x2b9e5f9a25e0 (unknown)
    @     0x2b9e602f21f7 __GI_raise
    @     0x2b9e602f38e8 __GI_abort
    @          0x30d708e starrocks::failure_function()
    @          0x63b8ffd google::LogMessage::Fail()
    @          0x63bb46f google::LogMessage::SendToLog()
    @          0x63b8b4e google::LogMessage::Flush()
    @          0x63bba79 google::LogMessageFatal::~LogMessageFatal()
    @          0x486a3c5 starrocks::TabletUpdates::_check_for_apply()
    @          0x487dfa6 starrocks::TabletUpdates::rowset_commit()
    @          0x4812f8e starrocks::Tablet::rowset_commit()
    @          0x489f653 starrocks::TxnManager::publish_txn()
    @          0x3096026 _ZZN9starrocks24run_publish_version_taskEPNS_15ThreadPoolTokenERKNS_22TPublishVersionRequestERNS_18TFinishTaskRequestERSt13unordered_setIPNS_7DataDirESt4hashIS9_ESt8equal_toIS9_ESaIS9_EEjENKUlvE_clEv
    @          0x51a4692 starrocks::ThreadPool::dispatch_thread()
    @          0x519f18a starrocks::Thread::supervise_thread()
    @     0x2b9e5f99ae25 start_thread
    @     0x2b9e603b534d __clone
    @                0x0 (unknown)

  • Github Issue:

  • Github Fix PR:

  • Jira

  • 问题版本:

  • 修复版本:

  • 问题原因:

    • 线程数超过 Linux 默认配置的线程数限制

  • 解决办法:

    • 修改 Linux 的线程数上限。
#188

  1. SinkBuffer user-after-free crash

*** Aborted at 1706246320 (unix time) try "date -d @1706246320" if you are using GNU date ***
PC: @          0x381da9c starrocks::pipeline::FragmentContext::cancel()
*** SIGSEGV (@0x17c) received by PID 2671 (TID 0x7f6673996700) from PID 380; stack trace: ***
    @          0x66c9be2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f670665f630 (unknown)
    @          0x381da9c starrocks::pipeline::FragmentContext::cancel()
    @          0x3aa8fdb _ZNSt17_Function_handlerIFvRKN9starrocks8pipeline14ClosureContextESt17basic_string_viewIcSt11char_traitsIcEEEZNS1_10SinkBuffer16_try_to_send_rpcERKNS0_9TUniqueIdERKSt8functionIFvvEEEUlS4_S8_E2_E9_M_invokeERKSt9_Any_dataS4_OS8_
    @          0x3ab098d starrocks::DisposableClosure<>::Run()
    @          0x689320b brpc::Controller::EndRPC()
    @          0x68934c0 brpc::Controller::RunEndRPC()
    @          0x683d26f bthread::TaskGroup::task_runner()
    @          0x6832a51 bthread_make_fcontext

或

*** Aborted at 1710438955 (unix time) try "date -d @1710438955" if you are using GNU date ***
PC: @          0x5d7cc80 (unknown)
*** SIGSEGV (@0x0) received by PID 97699 (TID 0x7f2a7927d700) from PID 0; stack trace: ***
    @          0x5c20f62 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f2cd1daa630 (unknown)
    @          0x5d7cc80 (unknown)
    @          0x5210cfb _ZZN9starrocks8pipeline10SinkBuffer16_try_to_send_rpcERKNS_9TUniqueIdERKSt8functionIFvvEEENKUlRKNS0_14ClosureContextERKNS_20PTransmitChunkResultEE3_clESC_SF_
    @          0x5215122 starrocks::DisposableClosure<>::Run()
    @          0x5db3b3b brpc::Controller::EndRPC()
    @          0x5e56826 brpc::policy::ProcessRpcResponse()
    @          0x5f38ac7 brpc::ProcessInputMessage()
    @          0x5f3999b brpc::InputMessenger::OnNewMessages()
    @          0x5ddf09e brpc::Socket::ProcessEvent()
    @          0x5d8309f bthread::TaskGroup::task_runner()
    @          0x5ec71c1 bthread_make_fcontext
be 频繁宕机
#189

  1. 同步物化视图查询 crash

*** Aborted at 1706602869 (unix time) try "date -d @1706602869" if you are using GNU date ***
PC: @          0x2fcdac8 starrocks::AggHashMapWithOneStringKeyWithNullable<>::compute_agg_noprefetch<>()
*** SIGSEGV (@0x1f7c) received by PID 66295 (TID 0x7f6ab9156700) from PID 8060; stack trace: ***
    @          0x6457542 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f6b5564d630 (unknown)
    @          0x2fcdac8 starrocks::AggHashMapWithOneStringKeyWithNullable<>::compute_agg_noprefetch<>()
    @          0x2f83689 (unknown)
    @          0x2f552ae starrocks::Aggregator::build_hash_map()
    @          0x2e6a7c2 starrocks::pipeline::AggregateBlockingSinkOperator::push_chunk()
    @          0x2b0246c starrocks::pipeline::PipelineDriver::process()
    @          0x5740e1e starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x500c91a starrocks::ThreadPool::dispatch_thread()
    @          0x50073aa starrocks::Thread::supervise_thread()
    @     0x7f6b55645ea5 start_thread
    @     0x7f6b54a46b0d __clone
    @                0x0 (unknown)
StarRocks执行挂载了同步物化视图的SQL后直接挂掉
#190

  1. export/select into outfile/写mysql 外表 Crash 或 卡住

*** Aborted at 1703119356 (unix time) try "date -d @1703119356" if you are using GNU date ***
PC: @          0x6581ac7 je_large_dalloc
*** SIGSEGV (@0x0) received by PID 44953 (TID 0x7ff180a2a700) from PID 0; stack trace: ***
    @          0x5b1ba42 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7ff28c0aee92 os::Linux::chained_handler()
    @     0x7ff28c0b5526 JVM_handle_linux_signal
    @     0x7ff28c0abb03 signalHandler()
    @     0x7ff28b57e5d0 (unknown)
    @          0x6581ac7 je_large_dalloc
    @          0x51285f8 std::_Rb_tree<>::_M_erase()
    @          0x51285b9 std::_Rb_tree<>::_M_erase()
    @          0x5129f21 starrocks::pipeline::ExportSinkIOBuffer::_open_file_writer()
    @          0x512a48f starrocks::pipeline::ExportSinkIOBuffer::_process_chunk()
    @          0x51282db starrocks::pipeline::SinkIOBuffer::execute_io_task()
    @          0x5c6252c bthread::ExecutionQueueBase::_execute()
    @          0x5c632a8 bthread::ExecutionQueueBase::_execute_tasks()
    @          0x494dff0 starrocks::PriorityThreadPool::work_thread()
    @          0x5adb407 thread_proxy
    @     0x7ff28b576dd5 start_thread
    @     0x7ff28ab9202d __clone
    @                0x0 (unknown)
BE节点宕机，je_large_dalloc
#191

  1. Brpc 报错 Resource temporarily unavailable

W0130 04:06:31.047230 1466553 disposable_closure.h:38] brpc failed, error=Resource temporarily unavailable, error_text=[E11]Resource temporarily unavailable
#192

  1. UDF mem_tracker crash

*** Aborted at 1686120661 (unix time) try "date -d @1686120661" if you are using GNU date ***
PC: @          0x2ce0b61 starrocks::MemTracker::consume()
*** SIGSEGV (@0x0) received by PID 117789 (TID 0x7f9cd0728700) from PID 0; stack trace: ***
    @          0x58f9dc2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f9d147ae362 (unknown)
    @     0x7f9d147b28b9 JVM_handle_linux_signal
    @     0x7f9d147a5f78 (unknown)
    @     0x7f9d13ce1390 (unknown)
    @          0x2ce0b61 starrocks::MemTracker::consume()
    @          0x486d328 _ZNSt17_Function_handlerIFvvEZN9starrocks24call_function_in_pthreadEPNS1_12RuntimeStateERKSt8functionIFNS1_6StatusEvEEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x475fbe0 starrocks::PriorityThreadPool::work_thread()
    @          0x58b97e7 thread_proxy
    @     0x7f9d13cd76ba start_thread
    @     0x7f9d132f841d clone
    @                0x0 (unknown)
#193

  1. 低基数导致查询报错

Dict Decode failed, Dict can't take cover all key :-424101658

  • Github Issue:

  • Github Fix PR:

  • Jira

  • 问题版本:

    • 3.1.0 ~ 3.1.7

    • 3.2.0 ~ 3.2.2

  • 修复版本:

    • 3.1.8+

    • 3.2.3+

  • 问题原因:

  • 临时解决办法:

    • set global cbo_enable_low_cardinality_optimize = false;
#194

  1. UnpackConstColumnBinaryFunction crash

*** Aborted at 1709171849 (unix time) try "date -d @1709171849" if you are using GNU date ***
PC: @          0x3b4efa9 starrocks::UnpackConstColumnBinaryFunction<>::evaluate<>()
*** SIGSEGV (@0x7f21cb212000) received by PID 3809085 (TID 0x7f284d7ef700) from PID 18446744072822530048; stack trace: ***
    @          0x62f3702 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f29dfb8e240 (unknown)
    @          0x3b4efa9 starrocks::UnpackConstColumnBinaryFunction<>::evaluate<>()
    @          0x3b500cc starrocks::VectorizedBinaryPredicate<>::evaluate_checked()
    @          0x3a0c283 starrocks::ExprContext::evaluate()
    @          0x3a0c5cf starrocks::ExprContext::evaluate()
    @          0x2aaadb2 starrocks::ExecNode::eval_conjuncts()
    @          0x2a973ac starrocks::pipeline::Operator::eval_conjuncts_and_in_filters()
    @          0x2d6c9a6 starrocks::pipeline::SelectOperator::push_chunk()
    @          0x2a6996b starrocks::pipeline::PipelineDriver::process()
    @          0x5607bbe starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x4f07072 starrocks::ThreadPool::dispatch_thread()
    @          0x4f01b6a starrocks::Thread::supervise_thread()
    @     0x7f29dfb83f2b (unknown)
    @     0x7f29df9286bf clone
    @                0x0 (unknown)

或

** Aborted at 1709285311 (unix time) try "date -d @1709285311" if you are using GNU date ***
PC: @          0x585822c starrocks::StringFunctions::replace()
*** SIGSEGV (@0x268377d514b6) received by PID 23198 (TID 0x7fe017715700) from PID 2010453174; stack trace: ***
    @          0x6457542 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fe227651630 (/usr/lib64/libpthread-2.17.so;65d5e4de (deleted)+0xf62f)
    @          0x585822c starrocks::StringFunctions::replace()
    @          0x438b554 starrocks::VectorizedFunctionCallExpr::evaluate_checked()
    @          0x43222c9 starrocks::DictMappingExpr::evaluate_checked()
    @          0x43c587e starrocks::VectorizedInConstPredicate<>::evaluate_with_filter()
    @          0x43c5c25 starrocks::VectorizedInConstPredicate<>::evaluate_checked()
    @          0x4386389 starrocks::VectorizedIfExpr<>::evaluate_checked()
    @          0x3aca163 starrocks::ExprContext::evaluate()
    @          0x442146c starrocks::RuntimeFilterProbeCollector::update_selectivity()
    @          0x442259f starrocks::RuntimeFilterProbeCollector::evaluate()
    @          0x2b2fb16 starrocks::pipeline::Operator::eval_runtime_bloom_filters()
    @          0x2b35c40 starrocks::pipeline::ScanOperator::pull_chunk()
    @          0x2b01b47 starrocks::pipeline::PipelineDriver::process()
    @          0x5740e1e starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x500c91a starrocks::ThreadPool::dispatch_thread()
    @          0x50073aa starrocks::Thread::supervise_thread()
    @     0x7fe227649ea5 (/usr/lib64/libpthread-2.17.so;65d5e4de (deleted)+0x7ea4)
    @     0x7fe226a4ab0d (/usr/lib64/libc-2.17.so;65d5e4de (deleted)+0xfeb0c)
    @                0x0 (unknown)

  • Github Issue:

  • Github Fix PR:

  • Jira

  • 问题版本:

    • 2.5.0 ~ 2.5.19

    • 3.0.0 ~ 3.0.9

    • 3.1.0 ~ 3.1.8

    • 3.2.0 ~ 3.2.3

  • 修复版本:

    • 2.5.20+

    • 2.0.10+

    • 3.1.9+

    • 3.2.4+

  • 问题原因:

  • 临时解决办法:

    • set global cbo_enable_low_cardinality_optimize = false;

    • set global cbo_push_down_distinct_below_window = false;

#195

  1. BE JVM 报错 Compressed class space

java.lang.OutOfMemoryError: Compressed class space

  • Github Issue:

  • Github Fix PR:

  • Jira

  • 问题版本:

    • 2.2.0 ~ latest

    • 2.3.0 ~ latest

    • 2.4.0 ~ latest

  • 修复版本:

    • 2.5.0+

  • 问题原因:

  • 临时解决办法:

    • 修改 be.conf 的 JAVA_OPTS 将 JVM 内存 限制调大
#196

  1. array_distinct crash

*** Aborted at 1700019561 (unix time) try “date -d @1700019561” if you are using GNU date ***
PC: @ 0x4f69240 starrocks::vectorized::ArrayDistinct<>::_array_distinct_item<>()
*** SIGSEGV (@0x7f2288c00000) received by PID 27723 (TID 0x7f5880903700) from PID 18446744071708868608; stack trace: ***
@ 0x5aed0a2 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f59d4dc9630 (unknown)
@ 0x4f69240 starrocks::vectorized::ArrayDistinct<>::_array_distinct_item<>()
@ 0x4f69a9a starrocks::vectorized::ArrayDistinct<>::_array_distinct<>()
@ 0x4f69f24 starrocks::vectorized::ArrayFunctions::array_distinct_varchar()
@ 0x3fe0ab4 starrocks::vectorized::VectorizedFunctionCallExpr::evaluate_checked()
@ 0x38ea553 starrocks::ExprContext::evaluate()
@ 0x38ea89f starrocks::ExprContext::evaluate()
@ 0x3009ac4 starrocks::pipeline::ProjectOperator::push_chunk()
@ 0x2d6ab2e starrocks::pipeline::PipelineDriver::process()
@ 0x510747a starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x4af5bc2 starrocks::ThreadPool::dispatch_thread()
@ 0x4af06ba starrocks:
:supervise_thread()
@ 0x7f59d4dc1ea5 start_thread
@ 0x7f59d43dc96d __clone
@ 0x0 (unknown)
#197

  1. 主键模型导入 crash

*** Aborted at 1709773243 (unix time) try “date -d @1709773243” if you are using GNU date ***
PC: @ 0x41fdae7 starrocks::ShardByLengthMutableIndex::memory_usage()
*** SIGSEGV (@0x0) received by PID 195386 (TID 0x7f4bc4020700) from PID 0; stack trace: ***
@ 0x5824342 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f535d7cd630 (unknown)
@ 0x41fdae7 starrocks::ShardByLengthMutableIndex::memory_usage()
@ 0x4069bd3 starrocks::UpdateManager::on_rowset_finished()
@ 0x467d357 starrocks::vectorized::DeltaWriter::commit()
@ 0x51019ab starrocks::vectorized::AsyncDeltaWriter::_execute()
@ 0x596ae2c bthread::ExecutionQueueBase::_execute()
@ 0x596bba8 bthread::ExecutionQueueBase::_execute_tasks()
@ 0x48c3b92 starrocks::ThreadPool::dispatch_thread()
@ 0x48be68a starrocks:
:supervise_thread()
@ 0x7f535d7c5ea5 start_thread
@ 0x7f535cde0b0d __clone
@ 0x0 (unknown)

或

*** Aborted at 1710326728 (unix time) try "date -d @1710326728" if you are using GNU date ***
PC: @          0x46f6714 starrocks::PrimaryIndex::memory_usage()
*** SIGSEGV (@0x88) received by PID 22619 (TID 0x7fb8b63f8700) from PID 136; stack trace: ***
    @          0x62e4042 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fb9481cb630 (unknown)
    @          0x46f6714 starrocks::PrimaryIndex::memory_usage()
    @          0x4810df6 starrocks::UpdateManager::on_rowset_finished()
    @          0x4e60687 starrocks::DeltaWriter::commit()
    @          0x49da1ce _ZNSt17_Function_handlerIFvvEZN9starrocks17SegmentFlushToken6submitEPN4brpc10ControllerEPKNS1_30PTabletWriterAddSegmentRequestEPNS1_29PTabletWriterAddSegmentResultEPN6google8protobuf7ClosureEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x50e93c2 starrocks::ThreadPool::dispatch_thread()
    @          0x50e3eba starrocks::Thread::supervise_thread()
    @     0x7fb9481c3ea5 start_thread
    @     0x7fb9477de96d __clone
    @                0x0 (unknown)
be 不定时宕机
#198

  1. 高频delete会导致be crash: version_for_delete_predicate

*** Aborted at 1710223298 (unix time) try “date -d @1710223298” if you are using GNU date ***
PC: @ 0x4591244 starrocks::TabletMeta::version_for_delete_predicate()
*** SIGSEGV (@0x2930000030a) received by PID 11091 (TID 0x7f0c54fff700) from PID 778; stack trace: ***
@ 0x63911c2 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f1b591e7630 (unknown)
@ 0x4591244 starrocks::TabletMeta::version_for_delete_predicate()
@ 0x4793b9a starrocks::SizeTieredCompactionPolicy::_pick_rowsets_to_size_tiered_compact()
@ 0x47968f0 starrocks::SizeTieredCompactionPolicy::need_compaction()
@ 0x4563e01 starrocks::Tablet::need_compaction()
@ 0x4788c3d starrocks::CompactionManager::update_tablet()
@ 0x4788f31 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager16_dispatch_workerEvEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x4f836b2 starrocks::ThreadPool::dispatch_thread()
@ 0x4f7e14a starrocks::supervise_thread()
@ 0x7f1b591dfea5 start_thread
@ 0x7f1b585e0b0d __clone
@ 0x0 (unknown)

  • Github Issue:

  • Github Fix PR:

  • Jira

  • 问题版本:

    • 2.5.0 ~ 2.5.19

    • 3.0.0 ~ 3.0.9

    • 3.1.0 ~ 3.1.9

    • 3.2.0 ~ 3.2.4

  • 修复版本:

    • 2.5.20+

    • 2.0.10+

    • 3.1.10+

    • 3.2.5+

  • 问题原因:

  • 临时解决办法:

    • be.conf enable_size_tiered_compaction_strategy=false
高频执行delete操作，BE节点异常退出
#199

  1. Plan check 报错

Type check failed. the type of arg 65: sum in expr ‘sum(65: sum)’ is defined as INT, but the actual type is BIGINT
at com.starrocks.sql.optimizer.validate.PlanValidator.validatePlan(PlanValidator.java:64) ~[starrocks-fe.jar:?]
at com.starrocks.sql.optimizer.Optimizer.optimizeByCost(Optimizer.java:235) ~[starrocks-fe.jar:?]
at com.starrocks.sql.optimizer.Optimizer.optimize(Optimizer.java:136) ~[starrocks-fe.jar:?]
at com.starrocks.sql.StatementPlanner.createQueryPlan(StatementPlanner.java:151) ~[starrocks-fe.jar:?]
at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:99) ~[starrocks-fe.jar:?]
at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:62) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:482) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:375) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:481) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:767) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
#200

  1. BE TabletSchemaMap 死锁

Thread 1331 (Thread 0x7f697b1fe700 (LWP 98875)):
#0  0x00007fe5a12794ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fe5a1274dcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007fe5a1274c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000042f5efb in starrocks::TabletSchemaMap::emplace(starrocks::TabletSchemaPB const&) ()
#4  0x00000000042d673d in starrocks::TabletMeta::init_from_pb(starrocks::TabletMetaPB*) ()
#5  0x00000000042b4e42 in starrocks::Tablet::generate_tablet_meta_copy_unlocked(std::shared_ptr<starrocks::TabletMeta> const&) const ()
#6  0x000000000429d05e in starrocks::SnapshotManager::snapshot_full[abi:cxx11](std::shared_ptr<starrocks::Tablet> const&, long, long, bool) ()
#7  0x000000000429f223 in starrocks::SnapshotManager::make_snapshot(starrocks::TSnapshotRequest const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) ()
#8  0x0000000002caccd1 in starrocks::AgentServer::Impl::make_snapshot(starrocks::TAgentResult&, starrocks::TSnapshotRequest const&) ()
#9  0x0000000004c4f94d in starrocks::BackendServiceProcessor::process_make_snapshot(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) ()
#10 0x0000000004c55c92 in starrocks::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, void*) ()
#11 0x0000000004c579a2 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) ()
#12 0x0000000005c081b8 in apache::thrift::server::TConnectedClient::run() ()
#13 0x0000000005c006b4 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() ()
#14 0x0000000005c02ebd in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) ()
#15 0x0000000005be8626 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread> > > >::_M_run() ()
#16 0x0000000008133fa0 in execute_native_thread_routine ()
#17 0x00007fe5a1272dd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fe5a088dead in clone () from /lib64/libc.so.6
#201

  1. Compaciton 大文件 crash

   @          0x5e50559 google::LogMessageFatal::~LogMessageFatal()
    @          0x27ecfaf starrocks::BinaryColumnBase<>::_build_slices()
    @          0x2757dd5 starrocks::BinaryColumnBase<>::raw_data()
    @          0x480a0da starrocks::ScalarColumnWriter::append()
    @          0x480a33c starrocks::StringColumnWriter::append()
    @          0x43a2806 starrocks::SegmentWriter::append_chunk()
    @          0x4a98da0 starrocks::VerticalRowsetWriter::add_columns()
    @          0x4a73ac7 starrocks::VerticalCompactionTask::_compact_data()
    @          0x4a74d0f starrocks::VerticalCompactionTask::_compact_column_group()
    @          0x4a75632 starrocks::VerticalCompactionTask::_vertical_compaction_data()
    @          0x4a76129 starrocks::VerticalCompactionTask::run_impl()
    @          0x4a6b6fc starrocks::CompactionTask::run()
    @          0x44d30d3 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager9_scheduleEvEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x4cc33d2 starrocks::ThreadPool::dispatch_thread()
    @          0x4cbde6a starrocks::Thread::supervise_thread()
    @     0x7f4658c6644b start_thread
    @     0x7f465803f40f __GI___clone
    @                0x0 (unknown)
#202

  1. Persistent index 导致 BE 启动失败

*** Aborted at 1711349618 (unix time) try "date -d @1711349618" if you are using GNU date ***
PC: @     0x7f39cefe5387 __GI_raise
*** SIGABRT (@0x3e80000c50f) received by PID 50447 (TID 0x7f39425ff700) from PID 50447; stack trace: ***
    @          0x5b1ba42 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f39cfa9a630 (unknown)
    @     0x7f39cefe5387 __GI_raise
    @     0x7f39cefe6a78 __GI_abort
    @          0x2cdf2be starrocks::failure_function()
    @          0x5b0f41d google::LogMessage::Fail()
    @          0x5b1188f google::LogMessage::SendToLog()
    @          0x5b0ef6e google::LogMessage::Flush()
    @          0x5b11e99 google::LogMessageFatal::~LogMessageFatal()
    @          0x4265eaf starrocks::TabletUpdates::_apply_rowset_commit()
    @          0x4266353 starrocks::TabletUpdates::do_apply()
    @          0x4b17465 starrocks::ThreadPool::dispatch_thread()
    @          0x4b11e4a starrocks::Thread::supervise_thread()
    @     0x7f39cfa92ea5 start_thread
    @     0x7f39cf0ad96d __clone
    @                0x0 (unknown)

W0325 14:58:06.988127 52911 rowset.cpp:141] Fail to open /data/starrocks/storage/be/data/1021/321209132/1248468755/0200000000005e48a648e0f47d7f27a81aa03e6bcc4b45b4_0.dat: Corruption: Bad segment file /data/starrocks/storage/be/data/1021/321209132/1248468755/0200000000005e48a648e0f47d7f27a81aa03e6bcc4b45b4_0.dat: file size 0 < 12
/build/starrocks/be/src/storage/rowset/segment.cpp:195 Segment::parse_segment_footer(read_file.get(), &footer, footer_length_hint, partial_rowset_footer)
/build/starrocks/be/src/storage/rowset/segment.cpp:67 segment->_open(footer_length_hint, partial_rowset_footer)
W0325 14:58:06.988687 52911 rowset_update_state.cpp:39] load RowsetUpdateState error: Corruption: Bad segment file /data/starrocks/storage/be/data/1021/321209132/1248468755/0200000000005e48a648e0f47d7f27a81aa03e6bcc4b45b4_0.dat: file size 0 < 12
/build/starrocks/be/src/storage/rowset/segment.cpp:195 Segment::parse_segment_footer(read_file.get(), &footer, footer_length_hint, partial_rowset_footer)
/build/starrocks/be/src/storage/rowset/segment.cpp:67 segment->_open(footer_length_hint, partial_rowset_footer)
/build/starrocks/be/src/storage/rowset/rowset.cpp:75 do_load()
/build/starrocks/be/src/storage/rowset/rowset.cpp:454 load()
/build/starrocks/be/src/storage/rowset_update_state.cpp:161 _load_upserts(rowset, 0, pk_column.get()) tablet:321209132 stack:
    @          0x46a4e19  _ZZSt9call_onceIZN9starrocks17RowsetUpdateState4loadEPNS0_6TabletEPNS0_6RowsetEEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv
    @     0x7fb3c0d7b20b  __pthread_once_slow
    @          0x469fde7  starrocks::RowsetUpdateState::load()
    @          0x4262183  starrocks::TabletUpdates::_apply_rowset_commit()
    @          0x4266353  starrocks::TabletUpdates::do_apply()
    @          0x4b17465  starrocks::ThreadPool::dispatch_thread()
    @          0x4b11e4a  starrocks::Thread::supervise_thread()
    @     0x7fb3c0d7cea5  start_thread
    @     0x7fb3c039796d  __clone
    @              (nil)  (unknown)

get_applied_rowsets failed, tablet updates is in error state: tablet:85018 actual row size changed after compaction 50531 -> 50041tablet:85018 #version:13 [29445 29456.1@12 29456.1] #pending:0 backend

F0423 22:31:29.743636 475679 tablet_updates.cpp:1132] delvec inconsistent tablet:8858730 rssid:5262 #old:1402 #add:4 #new:1402 old_v:10497 v:10498

  • Github Issue:

  • Github Fix PR:

  • Jira

  • 问题版本:

    • 2.5.0 ~ 2.5.20

    • 3.0.0 ~ 3.0.8

    • 3.1.0 ~ 3.1.5

  • 修复版本:

    • 2.5.21+

    • 3.0.9+

    • 3.1.6+

  • 问题原因:

  • 临时解决办法:

    • 使用 ./meta_tool.sh --operation=delete_persistent_index_meta 功能 删除有问题 tablet 的 persistent index 并重新启动 。如果是3副本的话可以使用 ./meta_tool.sh --operation=delete_meta功能删除
be节点异常宕机，无法拉起
compaction引起的be宕机问题
#203

京ICP备2022026421号-1