常见 Crash / BUG / 优化查询

trueeyu · 2023年01月17日 02:10

使用 Resource Group Crash

*** Aborted at 1664261214 (unix time) try "date -d @1664261214" if you are using GNU date ***
terminate called recursively
terminate called recursively
terminate called recursively
terminate called after throwing an instance of 'PC: @          0x29017b7 starrocks::workgroup::ScanExecutor::worker_thread()
*** SIGSEGV (@0x0) received by PID 30055 (TID 0x7fc381d46700) from PID 0; stack trace: ***
terminate called recursively
std::bad_function_call'
  what():  bad_function_call    @          0x3fa3ad2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fc3e0472630 (unknown)

    @          0x29017b7 starrocks::workgroup::ScanExecutor::worker_thread()
    @          0x2132549 starrocks::ThreadPool::dispatch_thread()
terminate called recursively
terminate called recursively
terminate called recursively
    @          0x212e0fa starrocks::Thread::supervise_thread()
    @     0x7fc3e046aea5 start_thread
    @     0x7fc3dfa859fd __clone
    @                0x0 (unknown)
start time: 2022年 09月 27日 星期二 14:50:00 CST

Github Issue: https://github.com/StarRocks/StarRocksTest/issues/1072
Github Fix PR: https://github.com/StarRocks/starrocks/pull/10983
问题版本: 2.3.0~2.3.2
修复版本: 2.3.3
临时规避方法: set global enable_resource_group = false;
问题原因: TODO

trueeyu · 2022年10月25日 11:09

主键模型 Partial update crash

start time: Fri Oct 21 17:42:12 CST 2022
*** Check failure stack trace: ***
    @          0x3cb1f8d  google::LogMessage::Fail()
    @          0x3cb43ff  google::LogMessage::SendToLog()
    @          0x3cb1ade  google::LogMessage::Flush()
    @          0x3cb4a09  google::LogMessageFatal::~LogMessageFatal()
    @          0x1a5415f  _ZN9starrocks26field_type_dispatch_columnINS_10vectorized13ColumnBuilderEJbEEEDaNS_9FieldTypeET_DpT0_
*** Check failure stack trace: ***
*** Check failure stack trace: ***
    @          0x1a46555  starrocks::vectorized::ChunkHelper::column_from_field_type()
    @          0x3cb1f8d  google::LogMessage::Fail()
    @          0x3cb1f8d  google::LogMessage::Fail()
    @          0x1b83f84  starrocks::RowsetUpdateState::_prepare_partial_update_states()
    @          0x3cb43ff  google::LogMessage::SendToLog()
    @          0x3cb43ff  google::LogMessage::SendToLog()
    @          0x1b85de5  starrocks::RowsetUpdateState::_do_load()
    @          0x1b85f65  _ZZSt9call_onceIZN9starrocks17RowsetUpdateState4loadEPNS0_6TabletEPNS0_6RowsetEEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv
*** Check failure stack trace: ***
    @          0x3cb1ade  google::LogMessage::Flush()
    @          0x3cb1ade  google::LogMessage::Flush()
    @          0x3cb1f8d  google::LogMessage::Fail()
    @          0x3cb4a09  google::LogMessageFatal::~LogMessageFatal()
    @          0x3cb43ff  google::LogMessage::SendToLog()
    @          0x3cb4a09  google::LogMessageFatal::~LogMessageFatal()
    @          0x1a5415f  _ZN9starrocks26field_type_dispatch_columnINS_10vectorized13ColumnBuilderEJbEEEDaNS_9FieldTypeET_DpT0_
    @          0x3cb1ade  google::LogMessage::Flush()
    @          0x1a5415f  _ZN9starrocks26field_type_dispatch_columnINS_10vectorized13ColumnBuilderEJbEEEDaNS_9FieldTypeET_DpT0_
    @          0x1a46555  starrocks::vectorized::ChunkHelper::column_from_field_type()
    @          0x3cb4a09  google::LogMessageFatal::~LogMessageFatal()
    @          0x1b83f84  starrocks::RowsetUpdateState::_prepare_partial_update_states()
    @          0x1a46555  starrocks::vectorized::ChunkHelper::column_from_field_type()
    @          0x1b85de5  starrocks::RowsetUpdateState::_do_load()
    @          0x1b85f65  _ZZSt9call_onceIZN9starrocks17RowsetUpdateState4loadEPNS0_6TabletEPNS0_6RowsetEEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv
    @          0x1b83f84  starrocks::RowsetUpdateState::_prepare_partial_update_states()
    @          0x1a5415f  _ZN9starrocks26field_type_dispatch_columnINS_10vectorized13ColumnBuilderEJbEEEDaNS_9FieldTypeET_DpT0_
    @          0x1b85de5  starrocks::RowsetUpdateState::_do_load()
    @          0x1b85f65  _ZZSt9call_onceIZN9starrocks17RowsetUpdateState4loadEPNS0_6TabletEPNS0_6RowsetEEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv
    @          0x1a46555  starrocks::vectorized::ChunkHelper::column_from_field_type()
    @          0x1b83f84  starrocks::RowsetUpdateState::_prepare_partial_update_states()
    @          0x1b85de5  starrocks::RowsetUpdateState::_do_load()
    @          0x1b85f65  _ZZSt9call_onceIZN9starrocks17RowsetUpdateState4loadEPNS0_6TabletEPNS0_6RowsetEEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv
    @     0x7f4fdc6b220b  __pthread_once_slow
    @          0x1b81fb3  starrocks::RowsetUpdateState::load()

BE 日志中有下面的错误:

F1025  array not supported

在2.2.5及之前的版本带有array类型表执行partial update时可能导致be crash

Github issue: https://github.com/StarRocks/starrocks/issues/10274
Github pr: https://github.com/StarRocks/starrocks/pull/10303
问题版本：2.2.0~2.2.5
修复版本: 2.2.6+
临时规避方法: 找到对应的tablet，truncate对应的partition，不使用partial update功能

trueeyu · 2022年11月3日 06:39

Base Compaction Crash

*** Aborted at 1666344928 (unix time) try date -d @1666344928 if you are using GNU date ***
               0xleb8ba0 starrocks::vectorized:：MaskMergeIterator::do_get_next()
PC: @                                                                                   ***
*** SIGSEGV  (@0x0) received by PID 173831 (TID 0x7f7e52a0e700) from PID 07 stack  trace:
     @         0x3ff7972 google::(anonymous namespace)::FailureSignalHandler()
     @    Ox7f7f4dlcd630 (unknown)
     @         0xleb8ba0 starrocks::vectorized:：MaskMergeIterator::do_get_next()
     @         0xlbe2cle starrocks::vectorized::TabletReader::do_get_next()
     @         0xlbfcfl9 starrocks::vectorized::Compaction::_merge_rowsets_vertically()
     @         0xlbfe7c2 starrocks::vectorized::Compaction::do_compaction_impl()
     @         0xlbff96b starrocks::vectorized::Compaction::do_compaction()
     @         0xlbf7f3e starrocks::vectorized::BaseCompaction::compact()
     @         0xla7258d starrocks::StorageEngine::_perform_base_compaction()
     @         0xlc4e2fe starrocks::StorageEngine::_base_compaction_thread_callback()
     @         0x5a752do execute_native_thread_routine
     @  0x7f7f4dlc5ea5 start_thread
     @  0x7f7f4c7e0b0d _clone
               0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/12906
Github Fix PR: https://github.com/StarRocks/starrocks/pull/12907
问题版本:
- 2.0.0 ~ 2.0.9
- 2.1.0 ~ 2.1.13
- 2.2.0 ~ 2.2.8
- 2.3.0 ~ 2.3.3
- 2.4.0
修复版本:
- 2.0.10
- 2.1.14
- 2.2.9
- 2.3.4
- 2.4.1
临时规避方法:
- 修改 be.conf max_consumer_num_per_group = 5000 (会增加Compaction 内存消耗)
问题原因:
- 在新加的列上，有删除条件

trueeyu · 2022年11月8日 11:02

Window_funnel crash

Aborted at 1667878017 (unix time) try "date -d @1667878017" if you are using GNU date *** PC:@
0x7føf43e16f4c __memcpy_ssse3_back
***
SIGSEGV (@0x79) received by PID 108628 (TID 0x7føe9d4fc700) from PID 121; stack trace:*** @
0x3f9døc2 google::(anonymous namespace)::FailureSignalHandler( ) @
0x7føf452ff5d0(unknown) @
0x7føf43e16f4c__memcpy_ssse3_back @

0x1ab0671 starrocks::vectorized::FixedLengthColumnBase<>::append() @
0x25a7e72 starrocks::vectorized::Nullablecolumn::append() @
0x2591bd2 starrocks::vectorized::ArrayColumn::append() @
0x2591dd1 starrocks::vectorized::ArrayColumn::append_selective() @
0x25a7d33 starrocks: :vectorized::Nullablecolumn::append_selective() @
0x2595dda starrocks::vectorized::Chunk::append_selective() @
0x3572970 starrocks::pipeline::Exchangesinkoperator::Channel::add_rows_selective() @
0x357429d starrocks::pipeline: :Exchangesinkoperator::push_chunk() @
0x2961e3c starrocks::pipeline::PipelineDriver::process() @
0x2957234 starrocks::pipeline::GlobalDriverExecutor::_worker_thread() @
0x21fd219 starrocks: :ThreadPool::dispatch_thread() @
0x21f8dca starrocks: :Thread: :supervise_thread()

Github Issue: https://github.com/StarRocks/starrocks/issues/9190
Github Fix PR: https://github.com/StarRocks/starrocks/pull/9191
问题版本:
- 2.3.0
修复版本:
- 2.3.1+
临时规避方法:
- 不使用漏斗函数
问题原因:
- Array 里面有 null

trueeyu · 2022年11月8日 11:19

查询中有 Like Crash

*** Aborted at 1667356261 (unix time) try "date -d @1667356261" if you are using GNU date ***
PC: @          0x4f3642b tcmalloc::ThreadCache::ReleaseToCentralCache()
*** SIGSEGV (@0x0) received by PID 7706 (TID 0x7f9632bc7700) from PID 0; stack trace: ***
    @          0x3592822 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f9650c20630 (unknown)
    @          0x4f3642b tcmalloc::ThreadCache::ReleaseToCentralCache()
    @          0x4f3672f tcmalloc::ThreadCache::Scavenge()
    @          0x184a6c2 std::_Sp_counted_ptr_inplace<>::_M_dispose()
    @          0x184aa3a std::_Sp_counted_ptr_inplace<>::_M_dispose()
    @          0x18b589a starrocks::vectorized::SegmentIterator::close()
    @          0x190ebe5 starrocks::vectorized::ProjectionIterator::close()
    @          0x1c81e87 starrocks::SegmentIteratorWrapper::close()
    @          0x1944f89 starrocks::vectorized::TimedChunkIterator::close()
    @          0x193e716 starrocks::vectorized::TabletReader::close()
    @          0x2542818 starrocks::vectorized::TabletScanner::close()
    @          0x228a7b6 starrocks::vectorized::OlapScanNode::_scanner_thread()
    @          0x1cab040 starrocks::PriorityThreadPool::work_thread()
    @          0x3533f47 thread_proxy
    @     0x7f9650c18ea5 start_thread
    @     0x7f965001d9fd __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/5983
Github Fix PR: https://github.com/StarRocks/starrocks/pull/6021
问题版本:
- 2.1.0 ~ 2.1.13
- 2.0.0 ~ 2.0.9
修复版本:
- 2.1.14+
- 2.0.10+
临时规避方法:
- 无
问题原因:
- 类似于这种SQL: select * from t1 where c1 = ‘xxx’ and c2 !=‘yyy’ and c3 like concat("%",‘0’,"%");

trueeyu · 2022年11月8日 12:04

主键模型 compaction crash

query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000
*** Aborted at 1666941018 (unix time) try "date -d @1666941018" if you are using GNU date ***
PC: @          0x226b325 starrocks::vectorized::MaskMergeIterator::do_get_next()
*** SIGSEGV (@0x0) received by PID 24658 (TID 0x7f5a42fff700) from PID 0; stack trace: ***
    @          0x481e332 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f5bd751f630 (unknown)
    @          0x226b325 starrocks::vectorized::MaskMergeIterator::do_get_next()
    @          0x1f9623c starrocks::vectorized::RowsetMergerImpl<>::_do_merge_vertically()
    @          0x1f97a93 starrocks::vectorized::RowsetMergerImpl<>::do_merge()
    @          0x1f8b12c starrocks::vectorized::compaction_merge_rowsets()
    @          0x1e74e9c starrocks::TabletUpdates::_do_compaction()
    @          0x1e75e8c starrocks::TabletUpdates::compaction()
    @          0x1dec879 starrocks::StorageEngine::_perform_update_compaction()
    @          0x1fed5be starrocks::StorageEngine::_update_compaction_thread_callback()
    @          0x631ac60 execute_native_thread_routine
    @     0x7f5bd7517ea5 start_thread
    @     0x7f5bd6b3296d __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/12856
Github Fix PR: https://github.com/StarRocks/starrocks/pull/12797
问题版本:
- 2.4.0
修复版本:
- 2.4.1+
临时规避方法:
- 修改 be.conf vertical_compaction_max_columns_per_group=1000
问题原因:
- 见 issue 描述

trueeyu · 2022年11月14日 10:09

建表超时报错

# 客户端报错
fail to create tablet timed out unfinished replicas xxx

# BE 这个时间有大量日志
I1111 10:04:33.745174 91674 tablet_manager.cpp:346] Dropping tablet 1328953
I1111 10:04:33.745180 91675 tablet_manager.cpp:346] Dropping tablet 1330983
I1111 10:04:33.745381 91674 tablet_manager.cpp:346] Dropping tablet 1331035
I1111 10:04:33.745388 91675 tablet_manager.cpp:346] Dropping tablet 46931
I1111 10:04:33.745560 91674 tablet_manager.cpp:346] Dropping tablet 1333065
I1111 10:04:33.745597 91675 tablet_manager.cpp:346] Dropping tablet 1326923
I1111 10:04:33.745762 91674 tablet_manager.cpp:346] Dropping tablet 1326975
I1111 10:04:33.745785 91675 tablet_manager.cpp:346] Dropping tablet 1329005
I1111 10:04:33.745963 91674 tablet_manager.cpp:346] Dropping tablet 1331091

Github Issue: https://github.com/StarRocks/starrocks/issues/3539
Github Fix PR: https://github.com/StarRocks/starrocks/pull/3571
问题版本:
- 2.0.0 ~ 2.0.x
- 2.1.0 ~ 2.1.x
修复版本:
- 2.2.0+
临时规避方法:
- 修改建表的超时时间
问题原因:
- DropTablet 和 CreateTablet 共用一个队列，当有大量 Drop 请求时，会导致 CreateTablet 请求排队

trueeyu · 2022年11月21日 08:09

Tablet 磁盘间迁移 Crash

*** Aborted at 1668992799 (unix time) try "date -d @1668992799" if you are using GNU date ***
PC: @          0x203e811 starrocks::RowsetMetaPB::MergeFrom()
*** SIGSEGV (@0x8) received by PID 131926 (TID 0x7f3b3e417700) from PID 8; stack trace: ***
    @          0x354a282 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f3c3451e630 (unknown)
    @          0x203e811 starrocks::RowsetMetaPB::MergeFrom()
    @          0x17c027f starrocks::TabletMeta::to_meta_pb()
    @          0x17c06f8 starrocks::TabletMeta::_save_meta()
    @          0x17c0cbd starrocks::TabletMeta::save_meta()
    @          0x179a7fb starrocks::Tablet::save_meta()
    @          0x17b0c3a starrocks::TabletManager::_drop_tablet_directly_unlocked()
    @          0x17b10f4 starrocks::TabletManager::_drop_tablet_unlocked()
    @          0x17b22a4 starrocks::TabletManager::_add_tablet_unlocked()
    @          0x17b3097 starrocks::TabletManager::load_tablet_from_meta()
    @          0x17b4100 starrocks::TabletManager::load_tablet_from_dir()
    @          0x2e86752 starrocks::EngineStorageMigrationTask::_storage_migrate()
    @          0x2e870c4 starrocks::EngineStorageMigrationTask::execute()
    @          0x1786cce starrocks::StorageEngine::execute_task()
    @          0x20e5cbf starrocks::TaskWorkerPool::_clone_worker_thread_callback()
    @          0x5027870 execute_native_thread_routine
    @     0x7f3c34516ea5 start_thread
    @     0x7f3c3391bb0d __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/10325
Github Fix PR: https://github.com/StarRocks/starrocks/pull/10355
问题版本:
- 2.0.0 ~ 2.0.x
- 2.1.0 ~ 2.1.x
- 2.2.0 ~ 2.2.5
- 2.3.0 ~ 2.3.1
修复版本:
- 2.2.6+
- 2.3.2+
临时规避方法:
- 重启
问题原因:
- Drop tablet 未添加锁

trueeyu · 2023年02月10日 13:14

Grouping Sets crash

*** Aborted at 1676008050 (unix time) try "date -d @1676008050" if you are using GNU date ***
PC: @          0x23081c5 starrocks::ExecNode::eval_filter_null_values()
*** SIGSEGV (@0x26) received by PID 93759 (TID 0x7f8442523700) from PID 38; stack trace: ***
    @          0x3caf752 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f84b2e80d62 os::Linux::chained_handler()
    @     0x7f84b2e873f6 JVM_handle_linux_signal
    @     0x7f84b2e7d9d3 signalHandler()
    @     0x7f84b23505f0 (unknown)
    @          0x23081c5 starrocks::ExecNode::eval_filter_null_values()
    @          0x25d42ac starrocks::vectorized::RepeatNode::get_next()
    @          0x23ff2e5 starrocks::vectorized::DistinctStreamingNode::get_next()
    @          0x1e8fdd3 starrocks::PlanFragmentExecutor::_get_next_internal_vectorized()
    @          0x1e9170e starrocks::PlanFragmentExecutor::_open_internal_vectorized()
    @          0x1e922f7 starrocks::PlanFragmentExecutor::open()
    @          0x1e25f4b starrocks::FragmentExecState::execute()
    @          0x1e2a68c starrocks::FragmentMgr::exec_actual()
    @          0x1e2aef1 _ZNSt17_Function_handlerIFvvEZN9starrocks11FragmentMgr18exec_plan_fragmentERKNS1_23TExecPlanFragmentParamsERKSt8functionIFvPNS1_20PlanFragmentExecutorEEESC_EUlvE_E9_M_invokeERKSt9_Any_data
    @          0x1f6cc39 starrocks::ThreadPool::dispatch_thread()
    @          0x1f687ea starrocks::Thread::supervise_thread()
    @     0x7f84b2348e65 start_thread
    @     0x7f84b196388d __clone
    @                0x0 (unknown)

*** Aborted at 1669159907 (unix time) try "date -d @1669159907" if you are using GNU date ***
PC: @          0x390ff25 starrocks::vectorized::VectorizedFunctionCallExpr::evaluate()
*** SIGSEGV (@0xfffffffffffffff8) received by PID 4352 (TID 0x7f22abeb0700) from PID 18446744073709551608; stack trace: ***
    @          0x4776152 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f233aadf630 (unknown)
    @          0x390ff25 starrocks::vectorized::VectorizedFunctionCallExpr::evaluate()
    @          0x3492fd4 starrocks::vectorized::VectorizedBinaryPredicate<>::evaluate()
    @          0x33bfc6c starrocks::ExprContext::evaluate()
    @          0x2a67641 starrocks::eager_prune_eval_conjuncts()
    @          0x2a69126 starrocks::ExecNode::eval_conjuncts()
    @          0x2dbb9cb starrocks::pipeline::Operator::eval_conjuncts_and_in_filters()
    @          0x2e543d2 starrocks::pipeline::RepeatOperator::pull_chunk()
    @          0x2e16943 starrocks::pipeline::PipelineDriver::process()
    @          0x2e0d183 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x262f855 starrocks::ThreadPool::dispatch_thread()
    @          0x262ad7a starrocks::Thread::supervise_thread()
    @     0x7f233aad7ea5 start_thread
    @     0x7f233a0f2b0d __clone
    @                0x0 (unknown)

Github Issue:
- https://github.com/StarRocks/starrocks/issues/12275
- https://github.com/StarRocks/starrocks/issues/13216
Github Fix PR: https://github.com/StarRocks/starrocks/pull/13217
问题版本:
- 2.0.0 ~ 2.0.x
- 2.1.0 ~ 2.1.x
- 2.2.0 ~ 2.2.9
- 2.3.0 ~ 2.3.4
- 2.4.0 ~ 2.4.1
修复版本:
- 2.2.10+
- 2.3.5+
- 2.4.2+
临时规避方法:
- 方法1: 不使用 grouping sets
- 方法2: set global enable_global_runtime_filter=false; (会影响查询性能)
问题原因:
- 见 issue 描述

trueeyu · 2022年12月9日 12:13

Java UDF Open Crash

*** SIGSEGV (@0x0) received by PID 1340 (TID 0x7f2266f5c700) from PID 0; stack trace: ***
    @          0x481e332 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f22c7900d45 os::Linux::chained_handler()
    @     0x7f22c7905b61 JVM_handle_linux_signal
    @     0x7f22c78f8a48 signalHandler()
    @     0x7f22c6daf630 (unknown)
    @     0x7f22c7af43f0 (unknown)
    @     0x7f22c765d03b java_lang_String::create_from_str()
    @     0x7f22c765d0c1 java_lang_String::create_oop_from_str()
    @     0x7f22c76c1a52 jni_NewStringUTF
    @          0x3b9e524 starrocks::vectorized::ClassLoader::init()
    @          0x3a51c61 _ZZN9starrocks10vectorized20JavaFunctionCallExpr4openEPNS_12RuntimeStateEPNS_11ExprContextEN13starrocks_udf15FunctionContext18FunctionStateScopeEENKUlvE_clEv
    @          0x3a52e55 _ZNSt17_Function_handlerIFN9starrocks6StatusEvEZNS0_10vectorized20JavaFunctionCallExpr4openEPNS0_12RuntimeStateEPNS0_11ExprContextEN13starrocks_udf15FunctionContext18FunctionStateScopeEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x3b959f1 starrocks::call_function_in_pthread()
    @          0x3a54541 starrocks::vectorized::JavaFunctionCallExpr::open()
    @          0x3456240 starrocks::Expr::open()
    @          0x34552dc starrocks::ExprContext::clone()
    @          0x2258e39 starrocks::vectorized::ColumnExprPredicate::_add_expr_ctx()
    @          0x2258f43 starrocks::vectorized::ColumnExprPredicate::ColumnExprPredicate()
    @          0x226e3f5 starrocks::vectorized::PredicateParser::parse_expr_ctx()
    @          0x2bd8e35 starrocks::vectorized::OlapScanConjunctsManager::get_column_predicates()
    @          0x2f6ec41 starrocks::pipeline::OlapChunkSource::_init_reader_params()
    @          0x2f702cb starrocks::pipeline::OlapChunkSource::_init_olap_reader()
    @          0x2f70bb4 starrocks::pipeline::OlapChunkSource::prepare()
    @          0x2e3b8d9 starrocks::pipeline::ScanOperator::_pickup_morsel()
    @          0x2e3bb35 starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan()
    @          0x2e3bd53 starrocks::pipeline::ScanOperator::pull_chunk()
    @          0x2e77eb3 starrocks::pipeline::PipelineDriver::process()
    @          0x2e6e5a3 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x2680a05 starrocks::ThreadPool::dispatch_thread()
    @          0x267bf2a starrocks::Thread::supervise_thread()
    @     0x7f22c6da7ea5 start_thread
    @     0x7f22c63c296d __clone

Github Issue: https://github.com/StarRocks/starrocks/issues/13787
Github Fix PR: https://github.com/StarRocks/starrocks/pull/13788
问题版本:
- 2.3.0 ~ 2.3.5
- 2.4.0 ~ 2.4.1
修复版本:
- 2.3.6+
- 2.4.2+
临时规避方法:
- 方法1: set global enable_column_expr_predicate=false; (会影响查询性能）
问题原因:
- 见 issue 描述

trueeyu · 2022年11月24日 12:13

Analyze too many versions

SHOW ANALYZE STATUS

有 Too many versions 错误

Github Issue: https://github.com/StarRocks/starrocks/issues/13261
Github Fix PR: https://github.com/StarRocks/starrocks/pull/13235
问题版本:
- 2.4.0 ~ 2.4.1
修复版本:
- 2.4.2+
临时规避方法:
- 无
问题原因:
- 2.4.0~2.4.1 统计信息收集频率变高

trueeyu · 2022年11月25日 03:13

RoaringBitmap Crash

*** Aborted at 1666613216 (unix time) try “date -d @1666613216” if you are using GNU date ***
PC: @ 0x4a4b44d roaring_read_uint32_iterator
*** SIGILL (@0x4a4b44d) received by PID 45161 (TID 0x7fac92d3c700) from PID 77902925; stack trace: ***
@ 0x45fa272 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7fad6111a630 (unknown)
@ 0x4a4b44d roaring_read_uint32_iterator
@ 0x1e95f5a starrocks::vectorized::roaring2range()
@ 0x1e9617d starrocks::vectorized::SegmentIterator::_apply_del_vector()
@ 0x1e9dcc2 starrocks::vectorized::SegmentIterator::_init()
@ 0x1e9e519 starrocks::vectorized::SegmentIterator::do_get_next()
@ 0x1ea8b02 starrocks::vectorized::SegmentIterator::do_get_next()
@ 0x1f44d12 starrocks::vectorized::MergeEntry<>::next()
@ 0x1f4819a starrocks::vectorized::RowsetMergerImpl<>::_do_merge_horizontally()
@ 0x1f49a6c starrocks::vectorized::RowsetMergerImpl<>::_do_merge_vertically()
@ 0x1f4b813 starrocks::vectorized::RowsetMergerImpl<>::do_merge()
@ 0x1f3f811 starrocks::vectorized::compaction_merge_rowsets()
@ 0x1e30d93 starrocks::TabletUpdates::_do_compaction()
@ 0x1e31d4c starrocks::TabletUpdates::compaction()
@ 0x1dab646 starrocks::StorageEngine::_perform_update_compaction()
@ 0x1f9fc4e starrocks::StorageEngine::_update_compaction_thread_callback()
@ 0x60f3a40 execute_native_thread_routine
@ 0x7fad61112ea5 start_thread
@ 0x7fad6072d8dd __clone
@ 0x0 (unknown)

临时规避方法:
- 换支持 AVX2 的机器
- 自已修改编译选项，编译一个不需要AVX2指令集的包
问题原因: 机器不支持 avx2

cat /proc/cpuinfo | grep avx2

trueeyu · 2022年11月25日 07:53

save meta crash

PC: @          0x203e811 starrocks::RowsetMetaPB::MergeFrom()
*** SIGSEGV (@0x8) received by PID 131926 (TID 0x7f3b3e417700) from PID 8; stack trace: ***
    @          0x354a282 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f3c3451e630 (unknown)
    @          0x203e811 starrocks::RowsetMetaPB::MergeFrom()
    @          0x17c027f starrocks::TabletMeta::to_meta_pb()
    @          0x17c06f8 starrocks::TabletMeta::_save_meta()
    @          0x17c0cbd starrocks::TabletMeta::save_meta()
    @          0x179a7fb starrocks::Tablet::save_meta()
    @          0x17b0c3a starrocks::TabletManager::_drop_tablet_directly_unlocked()
    @          0x17b10f4 starrocks::TabletManager::_drop_tablet_unlocked()
    @          0x17b22a4 starrocks::TabletManager::_add_tablet_unlocked()
    @          0x17b3097 starrocks::TabletManager::load_tablet_from_meta()
    @          0x17b4100 starrocks::TabletManager::load_tablet_from_dir()
    @          0x2e86752 starrocks::EngineStorageMigrationTask::_storage_migrate()
    @          0x2e870c4 starrocks::EngineStorageMigrationTask::execute()
    @          0x1786cce starrocks::StorageEngine::execute_task()
    @          0x20e5cbf starrocks::TaskWorkerPool::_clone_worker_thread_callback()
    @          0x5027870 execute_native_thread_routine
    @     0x7f3c34516ea5 start_thread
    @     0x7f3c3391bb0d __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/10325
Github Fix PR: https://github.com/StarRocks/starrocks/pull/10355
问题版本:
- 2.1.0 ~ 2.1.x
- 2.2.0 ~ 2.2.5
- 2.3.0 ~ 2.3.1
修复版本:
- 2.2.6+
- 2.3.2+
临时规避方法:
- 无
问题原因:
- save_meta 没加锁

trueeyu · 2022年11月28日 09:41

Sort Crash

PC: @          0x2520e16 starrocks::vectorized::HeapChunkSorter::_do_filter_data_for_type<>()
*** SIGSEGV (@0xc39) received by PID 300010 (TID 0x7fe795eba700) from PID 3129; stack trace: ***
    @          0x3cb75d2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fe83a5c5630 (unknown)
    @          0x2520e16 starrocks::vectorized::HeapChunkSorter::_do_filter_data_for_type<>()
    @          0x251792f starrocks::vectorized::HeapChunkSorter::_filter_data()
    @          0x2518e5d starrocks::vectorized::HeapChunkSorter::update()
    @          0x25138ec starrocks::vectorized::TopNNode::_consume_chunks()
    @          0x2514b6f starrocks::vectorized::TopNNode::open()
    @          0x1e948f4 starrocks::PlanFragmentExecutor::_open_internal_vectorized()
    @          0x1e955f7 starrocks::PlanFragmentExecutor::open()
    @          0x1e294bb starrocks::FragmentExecState::execute()
    @          0x1e2db9c starrocks::FragmentMgr::exec_actual()
    @          0x1e2e401 _ZNSt17_Function_handlerIFvvEZN9starrocks11FragmentMgr18exec_plan_fragmentERKNS1_23TExecPlanFragmentParamsERKSt8functionIFvPNS1_20PlanFragmentExecutorEEESC_EUlvE_E9_M_invokeERKSt9_Any_data
    @          0x1f723d9 starrocks::ThreadPool::dispatch_thread()
    @          0x1f6df8a starrocks::Thread::supervise_thread()
    @     0x7fe83a5bdea5 start_thread
    @     0x7fe839bd896d __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/10987
Github Fix PR: https://github.com/StarRocks/starrocks/pull/10988
问题版本:
- 2.3.0 ~ 2.3.2
修复版本:
- 2.3.3+
临时规避方法:
- 无
问题原因:
- 见 Issue 描述 (order by limit的时候，TopN处理有问题)

trueeyu · 2022年11月28日 09:47

Hudi 外表部分字段查询出来为Null

实际不是Null，但是查询出的结果中，部分字段为 null

Github Issue:
Github Fix PR: https://github.com/StarRocks/starrocks/pull/13058
问题版本:
- 2.3.0 ~ 2.3.4
- 2.4.0
修复版本:
- 2.3.5+
- 2.4.1+
临时规避方法:
- 修改be.conf: parquet_late_materialization_enable = false
问题原因:
- Parquet 延迟物化处理有问题

trueeyu · 2022年11月28日 09:57

主键模型表使用 persistent index 导致磁盘空间膨胀

主键模型表使用 persistent index, 导致磁盘空间不断增长，无法回收

Github Issue:
Github Fix PR: https://github.com/StarRocks/starrocks/pull/12862
问题版本:
- 2.3.0 ~ 2.3.4
- 2.4.0 ~ 2.4.1
修复版本:
- 2.3.5+
- 2.4.2+
临时规避方法:
- 无
问题原因:
- Persistent Index L0 index file 没有触发 compaction

trueeyu · 2022年11月28日 10:12

Ubuntu启动BE报错

be.out 报错

undefined symbol: _dl_sym, version GLIBC_PRIVATE

Github Issue: https://github.com/StarRocks/starrocks/issues/11035
Github Fix PR: https://github.com/StarRocks/starrocks/pull/14136
问题版本:
- 2.1.0 ~ 2.1.13
- 2.2.0 ~ 2.2.10
- 2.3.0 ~ 2.3.4
修复版本:
- 2.1.14+
- 2.2.11+
- 2.3.5+
临时规避方法:
- 无
问题原因:
- brpc库兼容高版本内核的问题

trueeyu · 2022年11月28日 10:50

Not Found string in global dict

Unique表，查询报错

Not Found string in global dict

Github Issue: https://github.com/StarRocks/starrocks/issues/13262
Github Fix PR: https://github.com/StarRocks/starrocks/pull/13263
问题版本:
- 2.1.0 ~ 2.1.x
- 2.2.0 ~ 2.2.9
- 2.3.0 ~ 2.3.4
- 2.4.0
修复版本:
- 2.2.10+
- 2.3.5+
- 2.4.1+
临时规避方法:
- set global cbo_enable_low_cardinality_optimize=false;
问题原因:
- 低基数优化处理有问题

trueeyu · 2022年12月1日 03:23

build hash map crash

0x7f83e010a6d0(unknown)
0x256c340 starrocks::Agguegator::build_hash_map<>()
0x28ac22b starrocks::pipeline::Aggregatestreamingsinkoperator::_push_chunk_by_auto()
0x28b2635 starrocks::pipeline::Aggregatestreamingsinkoperator::push_chunk()
0x289716c starrocks::pipeline::PipelineDriver::process()
0x288c564 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
0x2132549 starrocks::ThreadPool::dispatch_thread()
0x212e0fa starrocks::Thread::supervise_thread()
0x7f83e0102e25 start_thread
0x7f83df71ebad__clone
0x0(unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/9360
Github Fix PR: https://github.com/StarRocks/starrocks/pull/9548
问题版本:
- 2.2.0 ~ 2.2.10
- 2.3.0
修复版本:
- 2.2.11+
- 2.3.1+
临时规避方法:
- 无
问题原因:
- 执行计划有问题

trueeyu · 2022年12月2日 08:21

Schema Change+Add Partition+Balance之后，内存中的元数据出错

*** SIGABRT (@0x3e900000b7a) received by PID 2938 (TID 0x7efe4114a700) from PID 2938; stack trace: ***
    @          0x33a1be2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7efe9dc36630 (unknown)
    @     0x7efe9cf6b3d7 __GI_raise
    @     0x7efe9cf6cac8 __GI_abort
    @          0x157860d _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold
    @          0x4d3dfe6 __cxxabiv1::__terminate()
    @          0x4d3e051 std::terminate()
    @          0x4d3e1a5 __cxa_throw
    @          0x157a1f1 std::__throw_length_error()
    @          0x16ef0d0 std::vector<>::_M_fill_insert()
    @          0x16fa218 starrocks::vectorized::NullableColumn::append_numbers()
    @          0x1abe001 starrocks::segment_v2::BitShufflePageDecoder<>::next_batch()
    @          0x1ab43b7 starrocks::segment_v2::ParsedPageV2::read()
    @          0x2cae998 starrocks::segment_v2::ScalarColumnIterator::next_batch()
    @          0x2c173e1 starrocks::vectorized::SegmentIterator::_do_get_next()
    @          0x2c1ace1 starrocks::vectorized::SegmentIterator::do_get_next()
    @          0x1b0a40a starrocks::SegmentIteratorWrapper::do_get_next()
    @          0x189116b starrocks::vectorized::TimedChunkIterator::do_get_next()
    @          0x190cf3d starrocks::vectorized::HeapMergeIterator::do_get_next()
    @          0x189116b starrocks::vectorized::TimedChunkIterator::do_get_next()
    @          0x19041f1 starrocks::vectorized::AggregateIterator::do_get_next()
    @          0x189116b starrocks::vectorized::TimedChunkIterator::do_get_next()
    @          0x18c701a starrocks::vectorized::TabletReader::do_get_next()
    @          0x1822aa7 starrocks::vectorized::Compaction::merge_rowsets()
    @          0x18249b2 starrocks::vectorized::Compaction::do_compaction_impl()
    @          0x182679b starrocks::vectorized::Compaction::do_compaction()
    @          0x1820904 starrocks::vectorized::CumulativeCompaction::compact()
    @          0x1745439 starrocks::StorageEngine::_perform_cumulative_compaction()
    @          0x17326cf starrocks::StorageEngine::_cumulative_compaction_thread_callback()
    @          0x4db83a0 execute_native_thread_routine
    @     0x7efe9dc2eea5 start_thread
    @     0x7efe9d0339fd __clone

Github Issue: https://github.com/StarRocks/starrocks/issues/5433
Github Fix PR: https://github.com/StarRocks/starrocks/pull/4541q
问题版本:
- 2.0.0 ~ 2.0.3
- 2.1.0 ~ 2.1.3
修复版本:
- 2.0.4+
- 2.1.4+
临时规避方法:
- 如果出现之后，需要删表重建。出问题需要满足三个原因。一般是升级会触发Balance，导致最终出现问题。建议升级之前关闭Balance, 升级完成之后开启Balance

admin set frontend config ("max_scheduling_tablets"="0");
admin set frontend config ("disable_balance"="true");
admin set frontend config ("disable_colocate_balance"="true");

admin set frontend config ("max_scheduling_tablets"="2000");
admin set frontend config ("disable_balance"="false");
admin set frontend config ("disable_colocate_balance"="false");

问题原因:
- 存储层复用SchemaID，在上述操作之下出现错乱。

常见 Crash / BUG / 优化 查询