常见 Crash / BUG / 优化查询

trueeyu · 2024年01月26日 02:05

Export / Select into outfile / insert into files crash

*** Aborted at 1703119356 (unix time) try "date -d @1703119356" if you are using GNU date ***
PC: @          0x6581ac7 je_large_dalloc
*** SIGSEGV (@0x0) received by PID 44953 (TID 0x7ff180a2a700) from PID 0; stack trace: ***
    @          0x5b1ba42 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7ff28c0aee92 os::Linux::chained_handler()
    @     0x7ff28c0b5526 JVM_handle_linux_signal
    @     0x7ff28c0abb03 signalHandler()
    @     0x7ff28b57e5d0 (unknown)
    @          0x6581ac7 je_large_dalloc
    @          0x51285f8 std::_Rb_tree<>::_M_erase()
    @          0x51285b9 std::_Rb_tree<>::_M_erase()
    @          0x5129f21 starrocks::pipeline::ExportSinkIOBuffer::_open_file_writer()
    @          0x512a48f starrocks::pipeline::ExportSinkIOBuffer::_process_chunk()
    @          0x51282db starrocks::pipeline::SinkIOBuffer::execute_io_task()
    @          0x5c6252c bthread::ExecutionQueueBase::_execute()
    @          0x5c632a8 bthread::ExecutionQueueBase::_execute_tasks()
    @          0x494dff0 starrocks::PriorityThreadPool::work_thread()
    @          0x5adb407 thread_proxy
    @     0x7ff28b576dd5 start_thread
    @     0x7ff28ab9202d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
Jira
问题版本:
- 2.5 ~ latest
- 3.0 ~ latest
- 3.1 ~ latest
- 3.2 ~ latest
修复版本:
- 2.5 未修复
- 3.0 未修复
- 3.1 未修复
- 3.2 未修复
缓解版本:
- 2.5.19+
- 3.0.10+
- 3.1.0+
问题原因:
临时解决办法:

trueeyu · 2024年01月26日 12:15

大量删除 Tablet 后导致 BE 建表慢

有如下堆栈:

tid: 4925
         0x5ae583f  rocksdb::MemTableIterator::key()
         0x5afb41e  rocksdb::FragmentedRangeTombstoneList::FragmentTombstones()
         0x5afbc8f  rocksdb::FragmentedRangeTombstoneList::FragmentedRangeTombstoneList()
         0x5ae0376  rocksdb::MemTable::NewRangeTombstoneIterator()
         0x5ae0654  rocksdb::MemTable::Get()
         0x5a22f4f  rocksdb::DBImpl::GetImpl()
         0x5a23c41  rocksdb::DBImpl::Get()
         0x45731cd  starrocks::KVStore::get()
         0x42e7655  starrocks::TabletMetaManager::get_persistent_index_meta()
         0x433067b  starrocks::TabletUpdates::get_pk_index_write_amp_score()
         0x42cc631  starrocks::TabletManager::pick_tablets_to_do_pk_index_major_compaction()
         0x4555c37  starrocks::PersistentIndexCompactionManager::schedule()
         0x457c630  _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN9starrocks13StorageEngine16start_bg_threadsEvEUlvE5_EEEEE6_M_runEv
         0x7eda630  execute_native_thread_routine
    0x7f96359c5ea5  start_thread
    0x7f96356ee8dd  __clone
             (nil)  (unknown)
             
2 tids: 4844,4846
    0x7f96359c939c  __GI___pthread_rwlock_wrlock
         0x42d63df  starrocks::TabletManager::create_tablet()
         0x42838af  starrocks::StorageEngine::create_tablet()
         0x2747291  starrocks::run_create_tablet_task()
         0x4cc3e32  starrocks::ThreadPool::dispatch_thread()
         0x4cbe8ca  starrocks::Thread::supervise_thread()
    0x7f96359c5ea5  start_thread
    0x7f96356ee8dd  __clone
             (nil)  (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/38299
Jira
问题版本:
- 2.5.0 ~ 2.5.17
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.6
- 3.2.1 ~ 3.2.2
修复版本:
- 2.5.18+
- 3.0.10+
- 3.1.7+
- 3.2.3+
问题原因:
临时解决办法:

trueeyu · 2024年01月29日 06:07

同步物化视图 insert into crash

一般是物化视图的列中有大写字母会导致

*** Aborted at 1706016682 (unix time) try "date -d @1706016682" if you are using GNU date ***
PC: @          0x3a75a40 starrocks::BinaryColumnBase<>::append_selective()
*** SIGSEGV (@0x4) received by PID 31 (TID 0x7fe9dbffd640) from PID 4; stack trace: ***
    @          0x79bc8ea google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fea1a7d9896 os::Linux::chained_handler()
    @     0x7fea1a7df48b JVM_handle_linux_signal
    @     0x7fea1a7d20fc signalHandler()
    @     0x7fea1979b520 (unknown)
    @          0x3a75a40 starrocks::BinaryColumnBase<>::append_selective()
    @          0x60fac89 starrocks::MemTable::insert()
    @          0x60f03be starrocks::DeltaWriter::write()
    @          0x63cfebe starrocks::AsyncDeltaWriter::_execute()
    @          0x7b3a41c bthread::ExecutionQueueBase::_execute()
    @          0x7b3b298 bthread::ExecutionQueueBase::_execute_tasks()
    @          0x66202eb starrocks::ThreadPool::dispatch_thread()
    @          0x661a58a starrocks::Thread::supervise_thread()
    @     0x7fea197edb43 (unknown)
    @     0x7fea1987fa00 (unknown)
    @                0x0 (unknown)

Github Issue:
- https://github.com/StarRocks/starrocks/issues/31504
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/31505
Jira
问题版本:
- 3.1.0 ～ 3.1.3
修复版本:
- 3.1.4+
问题原因:
解决办法:
- 直接升级不能解决问题。需要删除有问题的同步物化视图，并升级到已修复版本。

trueeyu · 2024年01月29日 09:49

Union 报错: This sql plan has multi result sinks

一般有 CN 结点时出现

2024-01-26 11:04:20,527 WARN (starrocks-mysql-nio-pool-33031|282502) [StmtExecutor.execute():552] execute Exception, sql select 's'^M
union all^M
select org_no from dim.dim_org
com.starrocks.sql.common.StarRocksPlannerException: This sql plan has multi result sinks
        at com.starrocks.qe.Coordinator.computeFragmentExecParams(Coordinator.java:1702) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.Coordinator.prepareExec(Coordinator.java:640) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.Coordinator.exec(Coordinator.java:685) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:768) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:466) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:326) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:443) ~[starrocks-fe.jar:?]
        at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:710) ~[starrocks-fe.jar:?]
        at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:55) ~[starrocks-fe.jar:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_181]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_181]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_181]

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/25832
Jira
问题版本:
- 2.5.0 ～ 2.5.18
- 3.0.0 ～ 3.0.9
修复版本:
- 2.5.19+
- 3.0.10+
问题原因:
临时解决办法:
- set prefer_compute_node = false;

trueeyu · 2024年01月30日 03:27

DataStreamRecvr profile use-after-free

*** Aborted at 1700492373 (unix time) try "date -d @1700492373" if you are using GNU date ***
PC: @          0x32e5e30 starrocks::ScopedTimer<>::~ScopedTimer()
*** SIGSEGV (@0x0) received by PID 13416 (TID 0x7f4fa6d36700) from PID 0; stack trace: ***
    @          0x66b6ec2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f50274d0cfb os::Linux::chained_handler()
    @     0x7f50274d59bd JVM_handle_linux_signal
    @     0x7f50274c9ad8 signalHandler()
    @     0x7f5026691630 (unknown)
    @          0x32e5e30 starrocks::ScopedTimer<>::~ScopedTimer()
    @          0x5368545 starrocks::DataStreamRecvr::PipelineSenderQueue::try_to_build_chunk_meta()
    @          0x5372bbb starrocks::DataStreamRecvr::PipelineSenderQueue::add_chunks<>()
    @          0x536ab52 starrocks::DataStreamRecvr::PipelineSenderQueue::add_chunks()
    @          0x52ddbeb starrocks::DataStreamRecvr::add_chunks()
    @          0x527d63f starrocks::DataStreamMgr::transmit_chunk()
    @          0x5e107cc starrocks::PInternalServiceImplBase<>::_transmit_chunk()
    @          0x52a3600 starrocks::PriorityThreadPool::work_thread()
    @          0x6676887 thread_proxy
    @     0x7f5026689ea5 start_thread
    @     0x7f5025ca4b0d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/36780
Jira
问题版本:
- 2.5.0 ~ 2.5.16
- 3.0.0 ~ 3.0.8
- 3.1.0 ~ 3.1.5
- 3.2.0
修复版本:
- 2.5.17+
- 3.0.9+
- 3.1.6+
问题原因:
临时解决办法:

trueeyu · 2024年01月30日 05:59

Apply 线程数不够导致 crash

F1122 01:12:48.527740 150889 tablet_updates.cpp:788] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:432759625 #version:88 [30 98@86 98.1] pending: rowsets:1

*** Aborted at 1700375600 (unix time) try "date -d @1700375600" if you are using GNU date ***
PC: @     0x2b9e602f21f7 __GI_raise
*** SIGABRT (@0x7d00001111f) received by PID 69919 (TID 0x2ba20fb33700) from PID 69919; stack trace: ***
    @          0x63c5622 google::(anonymous namespace)::FailureSignalHandler()
    @     0x2b9e5f9a25e0 (unknown)
    @     0x2b9e602f21f7 __GI_raise
    @     0x2b9e602f38e8 __GI_abort
    @          0x30d708e starrocks::failure_function()
    @          0x63b8ffd google::LogMessage::Fail()
    @          0x63bb46f google::LogMessage::SendToLog()
    @          0x63b8b4e google::LogMessage::Flush()
    @          0x63bba79 google::LogMessageFatal::~LogMessageFatal()
    @          0x486a3c5 starrocks::TabletUpdates::_check_for_apply()
    @          0x487dfa6 starrocks::TabletUpdates::rowset_commit()
    @          0x4812f8e starrocks::Tablet::rowset_commit()
    @          0x489f653 starrocks::TxnManager::publish_txn()
    @          0x3096026 _ZZN9starrocks24run_publish_version_taskEPNS_15ThreadPoolTokenERKNS_22TPublishVersionRequestERNS_18TFinishTaskRequestERSt13unordered_setIPNS_7DataDirESt4hashIS9_ESt8equal_toIS9_ESaIS9_EEjENKUlvE_clEv
    @          0x51a4692 starrocks::ThreadPool::dispatch_thread()
    @          0x519f18a starrocks::Thread::supervise_thread()
    @     0x2b9e5f99ae25 start_thread
    @     0x2b9e603b534d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
Jira
问题版本:
修复版本:
问题原因:
- 线程数超过 Linux 默认配置的线程数限制
解决办法:
- 修改 Linux 的线程数上限。

trueeyu · 2024年05月14日 03:46

SinkBuffer user-after-free crash

*** Aborted at 1706246320 (unix time) try "date -d @1706246320" if you are using GNU date ***
PC: @          0x381da9c starrocks::pipeline::FragmentContext::cancel()
*** SIGSEGV (@0x17c) received by PID 2671 (TID 0x7f6673996700) from PID 380; stack trace: ***
    @          0x66c9be2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f670665f630 (unknown)
    @          0x381da9c starrocks::pipeline::FragmentContext::cancel()
    @          0x3aa8fdb _ZNSt17_Function_handlerIFvRKN9starrocks8pipeline14ClosureContextESt17basic_string_viewIcSt11char_traitsIcEEEZNS1_10SinkBuffer16_try_to_send_rpcERKNS0_9TUniqueIdERKSt8functionIFvvEEEUlS4_S8_E2_E9_M_invokeERKSt9_Any_dataS4_OS8_
    @          0x3ab098d starrocks::DisposableClosure<>::Run()
    @          0x689320b brpc::Controller::EndRPC()
    @          0x68934c0 brpc::Controller::RunEndRPC()
    @          0x683d26f bthread::TaskGroup::task_runner()
    @          0x6832a51 bthread_make_fcontext

或

*** Aborted at 1710438955 (unix time) try "date -d @1710438955" if you are using GNU date ***
PC: @          0x5d7cc80 (unknown)
*** SIGSEGV (@0x0) received by PID 97699 (TID 0x7f2a7927d700) from PID 0; stack trace: ***
    @          0x5c20f62 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f2cd1daa630 (unknown)
    @          0x5d7cc80 (unknown)
    @          0x5210cfb _ZZN9starrocks8pipeline10SinkBuffer16_try_to_send_rpcERKNS_9TUniqueIdERKSt8functionIFvvEEENKUlRKNS0_14ClosureContextERKNS_20PTransmitChunkResultEE3_clESC_SF_
    @          0x5215122 starrocks::DisposableClosure<>::Run()
    @          0x5db3b3b brpc::Controller::EndRPC()
    @          0x5e56826 brpc::policy::ProcessRpcResponse()
    @          0x5f38ac7 brpc::ProcessInputMessage()
    @          0x5f3999b brpc::InputMessenger::OnNewMessages()
    @          0x5ddf09e brpc::Socket::ProcessEvent()
    @          0x5d8309f bthread::TaskGroup::task_runner()
    @          0x5ec71c1 bthread_make_fcontext

Github Issue:
- https://github.com/StarRocks/StarRocksTest/issues/5911
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/40367
- https://github.com/StarRocks/starrocks/pull/42582
Jira
问题版本:
- 2.5.0 ~ 2.5.19
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.8
- 3.2.0 ~ 3.2.2
修复版本:
- 2.5.20+
- 3.0.10+
- 3.1.9+
- 3.2.3+
问题原因:
解决办法:

trueeyu · 2024年02月1日 07:48

同步物化视图查询 crash

*** Aborted at 1706602869 (unix time) try "date -d @1706602869" if you are using GNU date ***
PC: @          0x2fcdac8 starrocks::AggHashMapWithOneStringKeyWithNullable<>::compute_agg_noprefetch<>()
*** SIGSEGV (@0x1f7c) received by PID 66295 (TID 0x7f6ab9156700) from PID 8060; stack trace: ***
    @          0x6457542 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f6b5564d630 (unknown)
    @          0x2fcdac8 starrocks::AggHashMapWithOneStringKeyWithNullable<>::compute_agg_noprefetch<>()
    @          0x2f83689 (unknown)
    @          0x2f552ae starrocks::Aggregator::build_hash_map()
    @          0x2e6a7c2 starrocks::pipeline::AggregateBlockingSinkOperator::push_chunk()
    @          0x2b0246c starrocks::pipeline::PipelineDriver::process()
    @          0x5740e1e starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x500c91a starrocks::ThreadPool::dispatch_thread()
    @          0x50073aa starrocks::Thread::supervise_thread()
    @     0x7f6b55645ea5 start_thread
    @     0x7f6b54a46b0d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/40374
Jira
- https://starrocks.atlassian.net/browse/SR-23752
问题版本:
- 3.1.4~3.1.7
- 3.2.0~3.2.2
修复版本:
- 3.1.8+
- 3.2.3
问题原因:
临时解决办法:
- set global cbo_enable_low_cardinality_optimize=false;

trueeyu · 2024年02月18日 02:20

export/select into outfile/写mysql 外表 Crash 或卡住

*** Aborted at 1703119356 (unix time) try "date -d @1703119356" if you are using GNU date ***
PC: @          0x6581ac7 je_large_dalloc
*** SIGSEGV (@0x0) received by PID 44953 (TID 0x7ff180a2a700) from PID 0; stack trace: ***
    @          0x5b1ba42 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7ff28c0aee92 os::Linux::chained_handler()
    @     0x7ff28c0b5526 JVM_handle_linux_signal
    @     0x7ff28c0abb03 signalHandler()
    @     0x7ff28b57e5d0 (unknown)
    @          0x6581ac7 je_large_dalloc
    @          0x51285f8 std::_Rb_tree<>::_M_erase()
    @          0x51285b9 std::_Rb_tree<>::_M_erase()
    @          0x5129f21 starrocks::pipeline::ExportSinkIOBuffer::_open_file_writer()
    @          0x512a48f starrocks::pipeline::ExportSinkIOBuffer::_process_chunk()
    @          0x51282db starrocks::pipeline::SinkIOBuffer::execute_io_task()
    @          0x5c6252c bthread::ExecutionQueueBase::_execute()
    @          0x5c632a8 bthread::ExecutionQueueBase::_execute_tasks()
    @          0x494dff0 starrocks::PriorityThreadPool::work_thread()
    @          0x5adb407 thread_proxy
    @     0x7ff28b576dd5 start_thread
    @     0x7ff28ab9202d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/40674
- https://github.com/StarRocks/starrocks/pull/40874
Jira
问题版本:
- 2.5.0 ~ 2.5.19
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.8
- 3.2.0 ~ 3.2.3
修复版本:
- 2.5.20+
- 3.0.10+
- 3.1.9+
- 3.2.4+
问题原因:
临时解决办法:

trueeyu · 2024年03月26日 07:59

Brpc 报错 Resource temporarily unavailable

W0130 04:06:31.047230 1466553 disposable_closure.h:38] brpc failed, error=Resource temporarily unavailable, error_text=[E11]Resource temporarily unavailable

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/37962
- https://github.com/StarRocks/starrocks/pull/39285
Jira
- https://starrocks.atlassian.net/browse/SR-24742
问题版本:
- 2.5.0 ~ 2.5.19
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.8
- 3.2.0 ~ 3.2.3
修复版本:
- 2.5.20+
- 3.0.10+
- 3.1.9+
- 3.2.4+
问题原因:
临时解决办法:

trueeyu · 2024年02月27日 06:14

UDF mem_tracker crash

*** Aborted at 1686120661 (unix time) try "date -d @1686120661" if you are using GNU date ***
PC: @          0x2ce0b61 starrocks::MemTracker::consume()
*** SIGSEGV (@0x0) received by PID 117789 (TID 0x7f9cd0728700) from PID 0; stack trace: ***
    @          0x58f9dc2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f9d147ae362 (unknown)
    @     0x7f9d147b28b9 JVM_handle_linux_signal
    @     0x7f9d147a5f78 (unknown)
    @     0x7f9d13ce1390 (unknown)
    @          0x2ce0b61 starrocks::MemTracker::consume()
    @          0x486d328 _ZNSt17_Function_handlerIFvvEZN9starrocks24call_function_in_pthreadEPNS1_12RuntimeStateERKSt8functionIFNS1_6StatusEvEEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x475fbe0 starrocks::PriorityThreadPool::work_thread()
    @          0x58b97e7 thread_proxy
    @     0x7f9d13cd76ba start_thread
    @     0x7f9d132f841d clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/41710
Jira
问题版本:
- 2.5.0 ~ 2.5.19
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.8
- 3.2.0 ~ 3.2.3
修复版本:
- 2.5.20+
- 3.0.10+
- 3.1.9+
- 3.2.4+
问题原因:
临时解决办法:

trueeyu · 2024年03月11日 09:46

低基数导致查询报错

Dict Decode failed, Dict can't take cover all key :-424101658

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/40374
Jira
问题版本:
- 3.1.0 ~ 3.1.7
- 3.2.0 ~ 3.2.2
修复版本:
- 3.1.8+
- 3.2.3+
问题原因:
临时解决办法:
- set global cbo_enable_low_cardinality_optimize = false;

trueeyu · 2024年03月11日 09:46

UnpackConstColumnBinaryFunction crash

*** Aborted at 1709171849 (unix time) try "date -d @1709171849" if you are using GNU date ***
PC: @          0x3b4efa9 starrocks::UnpackConstColumnBinaryFunction<>::evaluate<>()
*** SIGSEGV (@0x7f21cb212000) received by PID 3809085 (TID 0x7f284d7ef700) from PID 18446744072822530048; stack trace: ***
    @          0x62f3702 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f29dfb8e240 (unknown)
    @          0x3b4efa9 starrocks::UnpackConstColumnBinaryFunction<>::evaluate<>()
    @          0x3b500cc starrocks::VectorizedBinaryPredicate<>::evaluate_checked()
    @          0x3a0c283 starrocks::ExprContext::evaluate()
    @          0x3a0c5cf starrocks::ExprContext::evaluate()
    @          0x2aaadb2 starrocks::ExecNode::eval_conjuncts()
    @          0x2a973ac starrocks::pipeline::Operator::eval_conjuncts_and_in_filters()
    @          0x2d6c9a6 starrocks::pipeline::SelectOperator::push_chunk()
    @          0x2a6996b starrocks::pipeline::PipelineDriver::process()
    @          0x5607bbe starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x4f07072 starrocks::ThreadPool::dispatch_thread()
    @          0x4f01b6a starrocks::Thread::supervise_thread()
    @     0x7f29dfb83f2b (unknown)
    @     0x7f29df9286bf clone
    @                0x0 (unknown)

或

** Aborted at 1709285311 (unix time) try "date -d @1709285311" if you are using GNU date ***
PC: @          0x585822c starrocks::StringFunctions::replace()
*** SIGSEGV (@0x268377d514b6) received by PID 23198 (TID 0x7fe017715700) from PID 2010453174; stack trace: ***
    @          0x6457542 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fe227651630 (/usr/lib64/libpthread-2.17.so;65d5e4de (deleted)+0xf62f)
    @          0x585822c starrocks::StringFunctions::replace()
    @          0x438b554 starrocks::VectorizedFunctionCallExpr::evaluate_checked()
    @          0x43222c9 starrocks::DictMappingExpr::evaluate_checked()
    @          0x43c587e starrocks::VectorizedInConstPredicate<>::evaluate_with_filter()
    @          0x43c5c25 starrocks::VectorizedInConstPredicate<>::evaluate_checked()
    @          0x4386389 starrocks::VectorizedIfExpr<>::evaluate_checked()
    @          0x3aca163 starrocks::ExprContext::evaluate()
    @          0x442146c starrocks::RuntimeFilterProbeCollector::update_selectivity()
    @          0x442259f starrocks::RuntimeFilterProbeCollector::evaluate()
    @          0x2b2fb16 starrocks::pipeline::Operator::eval_runtime_bloom_filters()
    @          0x2b35c40 starrocks::pipeline::ScanOperator::pull_chunk()
    @          0x2b01b47 starrocks::pipeline::PipelineDriver::process()
    @          0x5740e1e starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x500c91a starrocks::ThreadPool::dispatch_thread()
    @          0x50073aa starrocks::Thread::supervise_thread()
    @     0x7fe227649ea5 (/usr/lib64/libpthread-2.17.so;65d5e4de (deleted)+0x7ea4)
    @     0x7fe226a4ab0d (/usr/lib64/libc-2.17.so;65d5e4de (deleted)+0xfeb0c)
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/41740
Jira
问题版本:
- 2.5.0 ~ 2.5.19
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.8
- 3.2.0 ~ 3.2.3
修复版本:
- 2.5.20+
- 2.0.10+
- 3.1.9+
- 3.2.4+
问题原因:
临时解决办法:
- set global cbo_enable_low_cardinality_optimize = false;
- set global cbo_push_down_distinct_below_window = false;

trueeyu · 2024年03月5日 10:34

BE JVM 报错 Compressed class space

java.lang.OutOfMemoryError: Compressed class space

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/9944
Jira
问题版本:
- 2.2.0 ~ latest
- 2.3.0 ~ latest
- 2.4.0 ~ latest
修复版本:
- 2.5.0+
问题原因:
临时解决办法:
- 修改 be.conf 的 JAVA_OPTS 将 JVM 内存 限制调大

trueeyu · 2024年03月5日 12:33

array_distinct crash

*** Aborted at 1700019561 (unix time) try “date -d @1700019561” if you are using GNU date ***
PC: @ 0x4f69240 starrocks::vectorized::ArrayDistinct<>::_array_distinct_item<>()
*** SIGSEGV (@0x7f2288c00000) received by PID 27723 (TID 0x7f5880903700) from PID 18446744071708868608; stack trace: ***
@ 0x5aed0a2 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f59d4dc9630 (unknown)
@ 0x4f69240 starrocks::vectorized::ArrayDistinct<>::_array_distinct_item<>()
@ 0x4f69a9a starrocks::vectorized::ArrayDistinct<>::_array_distinct<>()
@ 0x4f69f24 starrocks::vectorized::ArrayFunctions::array_distinct_varchar()
@ 0x3fe0ab4 starrocks::vectorized::VectorizedFunctionCallExpr::evaluate_checked()
@ 0x38ea553 starrocks::ExprContext::evaluate()
@ 0x38ea89f starrocks::ExprContext::evaluate()
@ 0x3009ac4 starrocks::pipeline::ProjectOperator::push_chunk()
@ 0x2d6ab2e starrocks::pipeline::PipelineDriver::process()
@ 0x510747a starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x4af5bc2 starrocks::ThreadPool::dispatch_thread()
@ 0x4af06ba starrocks:
:supervise_thread()
@ 0x7f59d4dc1ea5 start_thread
@ 0x7f59d43dc96d __clone
@ 0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/36377
Jira
问题版本:
- 2.5.0 ~ 2.5.16
- 3.1.0 ~ 3.1.5
- 3.2.0
修复版本:
- 2.5.17+
- 3.1.6+
- 3.2.1+
问题原因:
临时解决办法:

trueeyu · 2024年05月23日 06:11

主键模型导入 crash

*** Aborted at 1709773243 (unix time) try “date -d @1709773243” if you are using GNU date ***
PC: @ 0x41fdae7 starrocks::ShardByLengthMutableIndex::memory_usage()
*** SIGSEGV (@0x0) received by PID 195386 (TID 0x7f4bc4020700) from PID 0; stack trace: ***
@ 0x5824342 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f535d7cd630 (unknown)
@ 0x41fdae7 starrocks::ShardByLengthMutableIndex::memory_usage()
@ 0x4069bd3 starrocks::UpdateManager::on_rowset_finished()
@ 0x467d357 starrocks::vectorized::DeltaWriter::commit()
@ 0x51019ab starrocks::vectorized::AsyncDeltaWriter::_execute()
@ 0x596ae2c bthread::ExecutionQueueBase::_execute()
@ 0x596bba8 bthread::ExecutionQueueBase::_execute_tasks()
@ 0x48c3b92 starrocks::ThreadPool::dispatch_thread()
@ 0x48be68a starrocks:
:supervise_thread()
@ 0x7f535d7c5ea5 start_thread
@ 0x7f535cde0b0d __clone
@ 0x0 (unknown)

或

*** Aborted at 1710326728 (unix time) try "date -d @1710326728" if you are using GNU date ***
PC: @          0x46f6714 starrocks::PrimaryIndex::memory_usage()
*** SIGSEGV (@0x88) received by PID 22619 (TID 0x7fb8b63f8700) from PID 136; stack trace: ***
    @          0x62e4042 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fb9481cb630 (unknown)
    @          0x46f6714 starrocks::PrimaryIndex::memory_usage()
    @          0x4810df6 starrocks::UpdateManager::on_rowset_finished()
    @          0x4e60687 starrocks::DeltaWriter::commit()
    @          0x49da1ce _ZNSt17_Function_handlerIFvvEZN9starrocks17SegmentFlushToken6submitEPN4brpc10ControllerEPKNS1_30PTabletWriterAddSegmentRequestEPNS1_29PTabletWriterAddSegmentResultEPN6google8protobuf7ClosureEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x50e93c2 starrocks::ThreadPool::dispatch_thread()
    @          0x50e3eba starrocks::Thread::supervise_thread()
    @     0x7fb9481c3ea5 start_thread
    @     0x7fb9477de96d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
Jira
问题版本:
- 2.5.0 ~ 2.5.20
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.10
- 3.2.0 ~ 3.2.6
修复版本:
- 2.5.21+
- 3.0.10+
- 3.1.11+
- 3.2.7+
问题原因:
临时解决办法:

trueeyu · 2024年05月20日 11:01

高频delete会导致be crash: version_for_delete_predicate

*** Aborted at 1710223298 (unix time) try “date -d @1710223298” if you are using GNU date ***
PC: @ 0x4591244 starrocks::TabletMeta::version_for_delete_predicate()
*** SIGSEGV (@0x2930000030a) received by PID 11091 (TID 0x7f0c54fff700) from PID 778; stack trace: ***
@ 0x63911c2 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f1b591e7630 (unknown)
@ 0x4591244 starrocks::TabletMeta::version_for_delete_predicate()
@ 0x4793b9a starrocks::SizeTieredCompactionPolicy::_pick_rowsets_to_size_tiered_compact()
@ 0x47968f0 starrocks::SizeTieredCompactionPolicy::need_compaction()
@ 0x4563e01 starrocks::Tablet::need_compaction()
@ 0x4788c3d starrocks::CompactionManager::update_tablet()
@ 0x4788f31 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager16_dispatch_workerEvEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x4f836b2 starrocks::ThreadPool::dispatch_thread()
@ 0x4f7e14a starrocks::supervise_thread()
@ 0x7f1b591dfea5 start_thread
@ 0x7f1b585e0b0d __clone
@ 0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/42100
- https://github.com/StarRocks/starrocks/pull/42174
Jira
问题版本:
- 2.5.0 ~ 2.5.19
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.9
- 3.2.0 ~ 3.2.4
修复版本:
- 2.5.20+
- 2.0.10+
- 3.1.10+
- 3.2.5+
问题原因:
临时解决办法:
- be.conf enable_size_tiered_compaction_strategy=false

trueeyu · 2024年03月18日 09:35

Plan check 报错

Type check failed. the type of arg 65: sum in expr ‘sum(65: sum)’ is defined as INT, but the actual type is BIGINT
at com.starrocks.sql.optimizer.validate.PlanValidator.validatePlan(PlanValidator.java:64) ~[starrocks-fe.jar:?]
at com.starrocks.sql.optimizer.Optimizer.optimizeByCost(Optimizer.java:235) ~[starrocks-fe.jar:?]
at com.starrocks.sql.optimizer.Optimizer.optimize(Optimizer.java:136) ~[starrocks-fe.jar:?]
at com.starrocks.sql.StatementPlanner.createQueryPlan(StatementPlanner.java:151) ~[starrocks-fe.jar:?]
at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:99) ~[starrocks-fe.jar:?]
at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:62) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:482) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:375) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:481) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:767) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/42454
Jira
问题版本:
- 3.1.0 ~ 3.1.9
- 3.2.0 ~ 3.2.4
修复版本:
- 3.1.10+
- 3.2.5+
问题原因:
临时解决办法:

trueeyu · 2024年03月18日 09:42

BE TabletSchemaMap 死锁

Thread 1331 (Thread 0x7f697b1fe700 (LWP 98875)):
#0  0x00007fe5a12794ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fe5a1274dcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007fe5a1274c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000042f5efb in starrocks::TabletSchemaMap::emplace(starrocks::TabletSchemaPB const&) ()
#4  0x00000000042d673d in starrocks::TabletMeta::init_from_pb(starrocks::TabletMetaPB*) ()
#5  0x00000000042b4e42 in starrocks::Tablet::generate_tablet_meta_copy_unlocked(std::shared_ptr<starrocks::TabletMeta> const&) const ()
#6  0x000000000429d05e in starrocks::SnapshotManager::snapshot_full[abi:cxx11](std::shared_ptr<starrocks::Tablet> const&, long, long, bool) ()
#7  0x000000000429f223 in starrocks::SnapshotManager::make_snapshot(starrocks::TSnapshotRequest const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) ()
#8  0x0000000002caccd1 in starrocks::AgentServer::Impl::make_snapshot(starrocks::TAgentResult&, starrocks::TSnapshotRequest const&) ()
#9  0x0000000004c4f94d in starrocks::BackendServiceProcessor::process_make_snapshot(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) ()
#10 0x0000000004c55c92 in starrocks::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, void*) ()
#11 0x0000000004c579a2 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) ()
#12 0x0000000005c081b8 in apache::thrift::server::TConnectedClient::run() ()
#13 0x0000000005c006b4 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() ()
#14 0x0000000005c02ebd in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) ()
#15 0x0000000005be8626 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread> > > >::_M_run() ()
#16 0x0000000008133fa0 in execute_native_thread_routine ()
#17 0x00007fe5a1272dd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fe5a088dead in clone () from /lib64/libc.so.6

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/40406
Jira
问题版本:
- 2.5.0 ～ 2.5.18
- 3.0.0 ～ 3.0.9
- 3.1.0 ～ 3.1.8
- 3.2.0 ～ 3.2.2
修复版本:
- 2.5.19+
- 3.0.10+
- 3.1.9+
- 3.2.3+
问题原因:
临时解决办法:

trueeyu · 2024年03月22日 11:15

Compaciton 大文件 crash

   @          0x5e50559 google::LogMessageFatal::~LogMessageFatal()
    @          0x27ecfaf starrocks::BinaryColumnBase<>::_build_slices()
    @          0x2757dd5 starrocks::BinaryColumnBase<>::raw_data()
    @          0x480a0da starrocks::ScalarColumnWriter::append()
    @          0x480a33c starrocks::StringColumnWriter::append()
    @          0x43a2806 starrocks::SegmentWriter::append_chunk()
    @          0x4a98da0 starrocks::VerticalRowsetWriter::add_columns()
    @          0x4a73ac7 starrocks::VerticalCompactionTask::_compact_data()
    @          0x4a74d0f starrocks::VerticalCompactionTask::_compact_column_group()
    @          0x4a75632 starrocks::VerticalCompactionTask::_vertical_compaction_data()
    @          0x4a76129 starrocks::VerticalCompactionTask::run_impl()
    @          0x4a6b6fc starrocks::CompactionTask::run()
    @          0x44d30d3 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager9_scheduleEvEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x4cc33d2 starrocks::ThreadPool::dispatch_thread()
    @          0x4cbde6a starrocks::Thread::supervise_thread()
    @     0x7f4658c6644b start_thread
    @     0x7f465803f40f __GI___clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/36281
Jira
问题版本:
- 2.5.0 ~ 2.5.20
- 3.0.0 ~ 3.0.8
- 3.1.0 ~ 3.1.5
- 3.2.0
修复版本:
- 2.5.21+
- 3.0.9+
- 3.1.7+
- 3.2.1+
问题原因:
临时解决办法:

常见 Crash / BUG / 优化 查询

Export / Select into outfile / insert into files crash

大量删除 Tablet 后导致 BE 建表慢

同步物化视图 insert into crash

Union 报错: This sql plan has multi result sinks

DataStreamRecvr profile use-after-free

Apply 线程数不够导致 crash

SinkBuffer user-after-free crash

同步物化视图查询 crash

export/select into outfile/写mysql 外表 Crash 或 卡住

Brpc 报错 Resource temporarily unavailable

UDF mem_tracker crash

低基数导致查询报错

UnpackConstColumnBinaryFunction crash

BE JVM 报错 Compressed class space

array_distinct crash

主键模型导入 crash

高频delete会导致be crash: version_for_delete_predicate

Plan check 报错

BE TabletSchemaMap 死锁

Compaciton 大文件 crash

常见 Crash / BUG / 优化查询

export/select into outfile/写mysql 外表 Crash 或卡住