常见 Crash / BUG / 优化查询

trueeyu · 2022年11月8日 11:02

Window_funnel crash

Aborted at 1667878017 (unix time) try "date -d @1667878017" if you are using GNU date *** PC:@
0x7føf43e16f4c __memcpy_ssse3_back
***
SIGSEGV (@0x79) received by PID 108628 (TID 0x7føe9d4fc700) from PID 121; stack trace:*** @
0x3f9døc2 google::(anonymous namespace)::FailureSignalHandler( ) @
0x7føf452ff5d0(unknown) @
0x7føf43e16f4c__memcpy_ssse3_back @

0x1ab0671 starrocks::vectorized::FixedLengthColumnBase<>::append() @
0x25a7e72 starrocks::vectorized::Nullablecolumn::append() @
0x2591bd2 starrocks::vectorized::ArrayColumn::append() @
0x2591dd1 starrocks::vectorized::ArrayColumn::append_selective() @
0x25a7d33 starrocks: :vectorized::Nullablecolumn::append_selective() @
0x2595dda starrocks::vectorized::Chunk::append_selective() @
0x3572970 starrocks::pipeline::Exchangesinkoperator::Channel::add_rows_selective() @
0x357429d starrocks::pipeline: :Exchangesinkoperator::push_chunk() @
0x2961e3c starrocks::pipeline::PipelineDriver::process() @
0x2957234 starrocks::pipeline::GlobalDriverExecutor::_worker_thread() @
0x21fd219 starrocks: :ThreadPool::dispatch_thread() @
0x21f8dca starrocks: :Thread: :supervise_thread()

Github Issue: https://github.com/StarRocks/starrocks/issues/9190
Github Fix PR: https://github.com/StarRocks/starrocks/pull/9191
问题版本:
- 2.3.0
修复版本:
- 2.3.1+
临时规避方法:
- 不使用漏斗函数
问题原因:
- Array 里面有 null

trueeyu · 2022年11月8日 11:19

查询中有 Like Crash

*** Aborted at 1667356261 (unix time) try "date -d @1667356261" if you are using GNU date ***
PC: @          0x4f3642b tcmalloc::ThreadCache::ReleaseToCentralCache()
*** SIGSEGV (@0x0) received by PID 7706 (TID 0x7f9632bc7700) from PID 0; stack trace: ***
    @          0x3592822 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f9650c20630 (unknown)
    @          0x4f3642b tcmalloc::ThreadCache::ReleaseToCentralCache()
    @          0x4f3672f tcmalloc::ThreadCache::Scavenge()
    @          0x184a6c2 std::_Sp_counted_ptr_inplace<>::_M_dispose()
    @          0x184aa3a std::_Sp_counted_ptr_inplace<>::_M_dispose()
    @          0x18b589a starrocks::vectorized::SegmentIterator::close()
    @          0x190ebe5 starrocks::vectorized::ProjectionIterator::close()
    @          0x1c81e87 starrocks::SegmentIteratorWrapper::close()
    @          0x1944f89 starrocks::vectorized::TimedChunkIterator::close()
    @          0x193e716 starrocks::vectorized::TabletReader::close()
    @          0x2542818 starrocks::vectorized::TabletScanner::close()
    @          0x228a7b6 starrocks::vectorized::OlapScanNode::_scanner_thread()
    @          0x1cab040 starrocks::PriorityThreadPool::work_thread()
    @          0x3533f47 thread_proxy
    @     0x7f9650c18ea5 start_thread
    @     0x7f965001d9fd __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/5983
Github Fix PR: https://github.com/StarRocks/starrocks/pull/6021
问题版本:
- 2.1.0 ~ 2.1.13
- 2.0.0 ~ 2.0.9
修复版本:
- 2.1.14+
- 2.0.10+
临时规避方法:
- 无
问题原因:
- 类似于这种SQL: select * from t1 where c1 = ‘xxx’ and c2 !=‘yyy’ and c3 like concat("%",‘0’,"%");

trueeyu · 2022年11月8日 12:04

主键模型 compaction crash

query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000
*** Aborted at 1666941018 (unix time) try "date -d @1666941018" if you are using GNU date ***
PC: @          0x226b325 starrocks::vectorized::MaskMergeIterator::do_get_next()
*** SIGSEGV (@0x0) received by PID 24658 (TID 0x7f5a42fff700) from PID 0; stack trace: ***
    @          0x481e332 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f5bd751f630 (unknown)
    @          0x226b325 starrocks::vectorized::MaskMergeIterator::do_get_next()
    @          0x1f9623c starrocks::vectorized::RowsetMergerImpl<>::_do_merge_vertically()
    @          0x1f97a93 starrocks::vectorized::RowsetMergerImpl<>::do_merge()
    @          0x1f8b12c starrocks::vectorized::compaction_merge_rowsets()
    @          0x1e74e9c starrocks::TabletUpdates::_do_compaction()
    @          0x1e75e8c starrocks::TabletUpdates::compaction()
    @          0x1dec879 starrocks::StorageEngine::_perform_update_compaction()
    @          0x1fed5be starrocks::StorageEngine::_update_compaction_thread_callback()
    @          0x631ac60 execute_native_thread_routine
    @     0x7f5bd7517ea5 start_thread
    @     0x7f5bd6b3296d __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/12856
Github Fix PR: https://github.com/StarRocks/starrocks/pull/12797
问题版本:
- 2.4.0
修复版本:
- 2.4.1+
临时规避方法:
- 修改 be.conf vertical_compaction_max_columns_per_group=1000
问题原因:
- 见 issue 描述

trueeyu · 2022年11月14日 10:09

建表超时报错

# 客户端报错
fail to create tablet timed out unfinished replicas xxx

# BE 这个时间有大量日志
I1111 10:04:33.745174 91674 tablet_manager.cpp:346] Dropping tablet 1328953
I1111 10:04:33.745180 91675 tablet_manager.cpp:346] Dropping tablet 1330983
I1111 10:04:33.745381 91674 tablet_manager.cpp:346] Dropping tablet 1331035
I1111 10:04:33.745388 91675 tablet_manager.cpp:346] Dropping tablet 46931
I1111 10:04:33.745560 91674 tablet_manager.cpp:346] Dropping tablet 1333065
I1111 10:04:33.745597 91675 tablet_manager.cpp:346] Dropping tablet 1326923
I1111 10:04:33.745762 91674 tablet_manager.cpp:346] Dropping tablet 1326975
I1111 10:04:33.745785 91675 tablet_manager.cpp:346] Dropping tablet 1329005
I1111 10:04:33.745963 91674 tablet_manager.cpp:346] Dropping tablet 1331091

Github Issue: https://github.com/StarRocks/starrocks/issues/3539
Github Fix PR: https://github.com/StarRocks/starrocks/pull/3571
问题版本:
- 2.0.0 ~ 2.0.x
- 2.1.0 ~ 2.1.x
修复版本:
- 2.2.0+
临时规避方法:
- 修改建表的超时时间
问题原因:
- DropTablet 和 CreateTablet 共用一个队列，当有大量 Drop 请求时，会导致 CreateTablet 请求排队

trueeyu · 2022年11月21日 08:09

Tablet 磁盘间迁移 Crash

*** Aborted at 1668992799 (unix time) try "date -d @1668992799" if you are using GNU date ***
PC: @          0x203e811 starrocks::RowsetMetaPB::MergeFrom()
*** SIGSEGV (@0x8) received by PID 131926 (TID 0x7f3b3e417700) from PID 8; stack trace: ***
    @          0x354a282 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f3c3451e630 (unknown)
    @          0x203e811 starrocks::RowsetMetaPB::MergeFrom()
    @          0x17c027f starrocks::TabletMeta::to_meta_pb()
    @          0x17c06f8 starrocks::TabletMeta::_save_meta()
    @          0x17c0cbd starrocks::TabletMeta::save_meta()
    @          0x179a7fb starrocks::Tablet::save_meta()
    @          0x17b0c3a starrocks::TabletManager::_drop_tablet_directly_unlocked()
    @          0x17b10f4 starrocks::TabletManager::_drop_tablet_unlocked()
    @          0x17b22a4 starrocks::TabletManager::_add_tablet_unlocked()
    @          0x17b3097 starrocks::TabletManager::load_tablet_from_meta()
    @          0x17b4100 starrocks::TabletManager::load_tablet_from_dir()
    @          0x2e86752 starrocks::EngineStorageMigrationTask::_storage_migrate()
    @          0x2e870c4 starrocks::EngineStorageMigrationTask::execute()
    @          0x1786cce starrocks::StorageEngine::execute_task()
    @          0x20e5cbf starrocks::TaskWorkerPool::_clone_worker_thread_callback()
    @          0x5027870 execute_native_thread_routine
    @     0x7f3c34516ea5 start_thread
    @     0x7f3c3391bb0d __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/10325
Github Fix PR: https://github.com/StarRocks/starrocks/pull/10355
问题版本:
- 2.0.0 ~ 2.0.x
- 2.1.0 ~ 2.1.x
- 2.2.0 ~ 2.2.5
- 2.3.0 ~ 2.3.1
修复版本:
- 2.2.6+
- 2.3.2+
临时规避方法:
- 重启
问题原因:
- Drop tablet 未添加锁

trueeyu · 2023年02月10日 13:14

Grouping Sets crash

*** Aborted at 1676008050 (unix time) try "date -d @1676008050" if you are using GNU date ***
PC: @          0x23081c5 starrocks::ExecNode::eval_filter_null_values()
*** SIGSEGV (@0x26) received by PID 93759 (TID 0x7f8442523700) from PID 38; stack trace: ***
    @          0x3caf752 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f84b2e80d62 os::Linux::chained_handler()
    @     0x7f84b2e873f6 JVM_handle_linux_signal
    @     0x7f84b2e7d9d3 signalHandler()
    @     0x7f84b23505f0 (unknown)
    @          0x23081c5 starrocks::ExecNode::eval_filter_null_values()
    @          0x25d42ac starrocks::vectorized::RepeatNode::get_next()
    @          0x23ff2e5 starrocks::vectorized::DistinctStreamingNode::get_next()
    @          0x1e8fdd3 starrocks::PlanFragmentExecutor::_get_next_internal_vectorized()
    @          0x1e9170e starrocks::PlanFragmentExecutor::_open_internal_vectorized()
    @          0x1e922f7 starrocks::PlanFragmentExecutor::open()
    @          0x1e25f4b starrocks::FragmentExecState::execute()
    @          0x1e2a68c starrocks::FragmentMgr::exec_actual()
    @          0x1e2aef1 _ZNSt17_Function_handlerIFvvEZN9starrocks11FragmentMgr18exec_plan_fragmentERKNS1_23TExecPlanFragmentParamsERKSt8functionIFvPNS1_20PlanFragmentExecutorEEESC_EUlvE_E9_M_invokeERKSt9_Any_data
    @          0x1f6cc39 starrocks::ThreadPool::dispatch_thread()
    @          0x1f687ea starrocks::Thread::supervise_thread()
    @     0x7f84b2348e65 start_thread
    @     0x7f84b196388d __clone
    @                0x0 (unknown)

*** Aborted at 1669159907 (unix time) try "date -d @1669159907" if you are using GNU date ***
PC: @          0x390ff25 starrocks::vectorized::VectorizedFunctionCallExpr::evaluate()
*** SIGSEGV (@0xfffffffffffffff8) received by PID 4352 (TID 0x7f22abeb0700) from PID 18446744073709551608; stack trace: ***
    @          0x4776152 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f233aadf630 (unknown)
    @          0x390ff25 starrocks::vectorized::VectorizedFunctionCallExpr::evaluate()
    @          0x3492fd4 starrocks::vectorized::VectorizedBinaryPredicate<>::evaluate()
    @          0x33bfc6c starrocks::ExprContext::evaluate()
    @          0x2a67641 starrocks::eager_prune_eval_conjuncts()
    @          0x2a69126 starrocks::ExecNode::eval_conjuncts()
    @          0x2dbb9cb starrocks::pipeline::Operator::eval_conjuncts_and_in_filters()
    @          0x2e543d2 starrocks::pipeline::RepeatOperator::pull_chunk()
    @          0x2e16943 starrocks::pipeline::PipelineDriver::process()
    @          0x2e0d183 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x262f855 starrocks::ThreadPool::dispatch_thread()
    @          0x262ad7a starrocks::Thread::supervise_thread()
    @     0x7f233aad7ea5 start_thread
    @     0x7f233a0f2b0d __clone
    @                0x0 (unknown)

Github Issue:
- https://github.com/StarRocks/starrocks/issues/12275
- https://github.com/StarRocks/starrocks/issues/13216
Github Fix PR: https://github.com/StarRocks/starrocks/pull/13217
问题版本:
- 2.0.0 ~ 2.0.x
- 2.1.0 ~ 2.1.x
- 2.2.0 ~ 2.2.9
- 2.3.0 ~ 2.3.4
- 2.4.0 ~ 2.4.1
修复版本:
- 2.2.10+
- 2.3.5+
- 2.4.2+
临时规避方法:
- 方法1: 不使用 grouping sets
- 方法2: set global enable_global_runtime_filter=false; (会影响查询性能)
问题原因:
- 见 issue 描述

trueeyu · 2022年12月9日 12:13

Java UDF Open Crash

*** SIGSEGV (@0x0) received by PID 1340 (TID 0x7f2266f5c700) from PID 0; stack trace: ***
    @          0x481e332 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f22c7900d45 os::Linux::chained_handler()
    @     0x7f22c7905b61 JVM_handle_linux_signal
    @     0x7f22c78f8a48 signalHandler()
    @     0x7f22c6daf630 (unknown)
    @     0x7f22c7af43f0 (unknown)
    @     0x7f22c765d03b java_lang_String::create_from_str()
    @     0x7f22c765d0c1 java_lang_String::create_oop_from_str()
    @     0x7f22c76c1a52 jni_NewStringUTF
    @          0x3b9e524 starrocks::vectorized::ClassLoader::init()
    @          0x3a51c61 _ZZN9starrocks10vectorized20JavaFunctionCallExpr4openEPNS_12RuntimeStateEPNS_11ExprContextEN13starrocks_udf15FunctionContext18FunctionStateScopeEENKUlvE_clEv
    @          0x3a52e55 _ZNSt17_Function_handlerIFN9starrocks6StatusEvEZNS0_10vectorized20JavaFunctionCallExpr4openEPNS0_12RuntimeStateEPNS0_11ExprContextEN13starrocks_udf15FunctionContext18FunctionStateScopeEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x3b959f1 starrocks::call_function_in_pthread()
    @          0x3a54541 starrocks::vectorized::JavaFunctionCallExpr::open()
    @          0x3456240 starrocks::Expr::open()
    @          0x34552dc starrocks::ExprContext::clone()
    @          0x2258e39 starrocks::vectorized::ColumnExprPredicate::_add_expr_ctx()
    @          0x2258f43 starrocks::vectorized::ColumnExprPredicate::ColumnExprPredicate()
    @          0x226e3f5 starrocks::vectorized::PredicateParser::parse_expr_ctx()
    @          0x2bd8e35 starrocks::vectorized::OlapScanConjunctsManager::get_column_predicates()
    @          0x2f6ec41 starrocks::pipeline::OlapChunkSource::_init_reader_params()
    @          0x2f702cb starrocks::pipeline::OlapChunkSource::_init_olap_reader()
    @          0x2f70bb4 starrocks::pipeline::OlapChunkSource::prepare()
    @          0x2e3b8d9 starrocks::pipeline::ScanOperator::_pickup_morsel()
    @          0x2e3bb35 starrocks::pipeline::ScanOperator::_try_to_trigger_next_scan()
    @          0x2e3bd53 starrocks::pipeline::ScanOperator::pull_chunk()
    @          0x2e77eb3 starrocks::pipeline::PipelineDriver::process()
    @          0x2e6e5a3 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x2680a05 starrocks::ThreadPool::dispatch_thread()
    @          0x267bf2a starrocks::Thread::supervise_thread()
    @     0x7f22c6da7ea5 start_thread
    @     0x7f22c63c296d __clone

Github Issue: https://github.com/StarRocks/starrocks/issues/13787
Github Fix PR: https://github.com/StarRocks/starrocks/pull/13788
问题版本:
- 2.3.0 ~ 2.3.5
- 2.4.0 ~ 2.4.1
修复版本:
- 2.3.6+
- 2.4.2+
临时规避方法:
- 方法1: set global enable_column_expr_predicate=false; (会影响查询性能）
问题原因:
- 见 issue 描述

trueeyu · 2022年11月24日 12:13

Analyze too many versions

SHOW ANALYZE STATUS

有 Too many versions 错误

Github Issue: https://github.com/StarRocks/starrocks/issues/13261
Github Fix PR: https://github.com/StarRocks/starrocks/pull/13235
问题版本:
- 2.4.0 ~ 2.4.1
修复版本:
- 2.4.2+
临时规避方法:
- 无
问题原因:
- 2.4.0~2.4.1 统计信息收集频率变高

trueeyu · 2022年11月25日 03:13

RoaringBitmap Crash

*** Aborted at 1666613216 (unix time) try “date -d @1666613216” if you are using GNU date ***
PC: @ 0x4a4b44d roaring_read_uint32_iterator
*** SIGILL (@0x4a4b44d) received by PID 45161 (TID 0x7fac92d3c700) from PID 77902925; stack trace: ***
@ 0x45fa272 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7fad6111a630 (unknown)
@ 0x4a4b44d roaring_read_uint32_iterator
@ 0x1e95f5a starrocks::vectorized::roaring2range()
@ 0x1e9617d starrocks::vectorized::SegmentIterator::_apply_del_vector()
@ 0x1e9dcc2 starrocks::vectorized::SegmentIterator::_init()
@ 0x1e9e519 starrocks::vectorized::SegmentIterator::do_get_next()
@ 0x1ea8b02 starrocks::vectorized::SegmentIterator::do_get_next()
@ 0x1f44d12 starrocks::vectorized::MergeEntry<>::next()
@ 0x1f4819a starrocks::vectorized::RowsetMergerImpl<>::_do_merge_horizontally()
@ 0x1f49a6c starrocks::vectorized::RowsetMergerImpl<>::_do_merge_vertically()
@ 0x1f4b813 starrocks::vectorized::RowsetMergerImpl<>::do_merge()
@ 0x1f3f811 starrocks::vectorized::compaction_merge_rowsets()
@ 0x1e30d93 starrocks::TabletUpdates::_do_compaction()
@ 0x1e31d4c starrocks::TabletUpdates::compaction()
@ 0x1dab646 starrocks::StorageEngine::_perform_update_compaction()
@ 0x1f9fc4e starrocks::StorageEngine::_update_compaction_thread_callback()
@ 0x60f3a40 execute_native_thread_routine
@ 0x7fad61112ea5 start_thread
@ 0x7fad6072d8dd __clone
@ 0x0 (unknown)

临时规避方法:
- 换支持 AVX2 的机器
- 自已修改编译选项，编译一个不需要AVX2指令集的包
问题原因: 机器不支持 avx2

cat /proc/cpuinfo | grep avx2

trueeyu · 2022年11月25日 07:53

save meta crash

PC: @          0x203e811 starrocks::RowsetMetaPB::MergeFrom()
*** SIGSEGV (@0x8) received by PID 131926 (TID 0x7f3b3e417700) from PID 8; stack trace: ***
    @          0x354a282 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f3c3451e630 (unknown)
    @          0x203e811 starrocks::RowsetMetaPB::MergeFrom()
    @          0x17c027f starrocks::TabletMeta::to_meta_pb()
    @          0x17c06f8 starrocks::TabletMeta::_save_meta()
    @          0x17c0cbd starrocks::TabletMeta::save_meta()
    @          0x179a7fb starrocks::Tablet::save_meta()
    @          0x17b0c3a starrocks::TabletManager::_drop_tablet_directly_unlocked()
    @          0x17b10f4 starrocks::TabletManager::_drop_tablet_unlocked()
    @          0x17b22a4 starrocks::TabletManager::_add_tablet_unlocked()
    @          0x17b3097 starrocks::TabletManager::load_tablet_from_meta()
    @          0x17b4100 starrocks::TabletManager::load_tablet_from_dir()
    @          0x2e86752 starrocks::EngineStorageMigrationTask::_storage_migrate()
    @          0x2e870c4 starrocks::EngineStorageMigrationTask::execute()
    @          0x1786cce starrocks::StorageEngine::execute_task()
    @          0x20e5cbf starrocks::TaskWorkerPool::_clone_worker_thread_callback()
    @          0x5027870 execute_native_thread_routine
    @     0x7f3c34516ea5 start_thread
    @     0x7f3c3391bb0d __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/10325
Github Fix PR: https://github.com/StarRocks/starrocks/pull/10355
问题版本:
- 2.1.0 ~ 2.1.x
- 2.2.0 ~ 2.2.5
- 2.3.0 ~ 2.3.1
修复版本:
- 2.2.6+
- 2.3.2+
临时规避方法:
- 无
问题原因:
- save_meta 没加锁

trueeyu · 2022年11月28日 09:41

Sort Crash

PC: @          0x2520e16 starrocks::vectorized::HeapChunkSorter::_do_filter_data_for_type<>()
*** SIGSEGV (@0xc39) received by PID 300010 (TID 0x7fe795eba700) from PID 3129; stack trace: ***
    @          0x3cb75d2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fe83a5c5630 (unknown)
    @          0x2520e16 starrocks::vectorized::HeapChunkSorter::_do_filter_data_for_type<>()
    @          0x251792f starrocks::vectorized::HeapChunkSorter::_filter_data()
    @          0x2518e5d starrocks::vectorized::HeapChunkSorter::update()
    @          0x25138ec starrocks::vectorized::TopNNode::_consume_chunks()
    @          0x2514b6f starrocks::vectorized::TopNNode::open()
    @          0x1e948f4 starrocks::PlanFragmentExecutor::_open_internal_vectorized()
    @          0x1e955f7 starrocks::PlanFragmentExecutor::open()
    @          0x1e294bb starrocks::FragmentExecState::execute()
    @          0x1e2db9c starrocks::FragmentMgr::exec_actual()
    @          0x1e2e401 _ZNSt17_Function_handlerIFvvEZN9starrocks11FragmentMgr18exec_plan_fragmentERKNS1_23TExecPlanFragmentParamsERKSt8functionIFvPNS1_20PlanFragmentExecutorEEESC_EUlvE_E9_M_invokeERKSt9_Any_data
    @          0x1f723d9 starrocks::ThreadPool::dispatch_thread()
    @          0x1f6df8a starrocks::Thread::supervise_thread()
    @     0x7fe83a5bdea5 start_thread
    @     0x7fe839bd896d __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/10987
Github Fix PR: https://github.com/StarRocks/starrocks/pull/10988
问题版本:
- 2.3.0 ~ 2.3.2
修复版本:
- 2.3.3+
临时规避方法:
- 无
问题原因:
- 见 Issue 描述 (order by limit的时候，TopN处理有问题)

trueeyu · 2022年11月28日 09:47

Hudi 外表部分字段查询出来为Null

实际不是Null，但是查询出的结果中，部分字段为 null

Github Issue:
Github Fix PR: https://github.com/StarRocks/starrocks/pull/13058
问题版本:
- 2.3.0 ~ 2.3.4
- 2.4.0
修复版本:
- 2.3.5+
- 2.4.1+
临时规避方法:
- 修改be.conf: parquet_late_materialization_enable = false
问题原因:
- Parquet 延迟物化处理有问题

trueeyu · 2022年11月28日 09:57

主键模型表使用 persistent index 导致磁盘空间膨胀

主键模型表使用 persistent index, 导致磁盘空间不断增长，无法回收

Github Issue:
Github Fix PR: https://github.com/StarRocks/starrocks/pull/12862
问题版本:
- 2.3.0 ~ 2.3.4
- 2.4.0 ~ 2.4.1
修复版本:
- 2.3.5+
- 2.4.2+
临时规避方法:
- 无
问题原因:
- Persistent Index L0 index file 没有触发 compaction

trueeyu · 2022年11月28日 10:12

Ubuntu启动BE报错

be.out 报错

undefined symbol: _dl_sym, version GLIBC_PRIVATE

Github Issue: https://github.com/StarRocks/starrocks/issues/11035
Github Fix PR: https://github.com/StarRocks/starrocks/pull/14136
问题版本:
- 2.1.0 ~ 2.1.13
- 2.2.0 ~ 2.2.10
- 2.3.0 ~ 2.3.4
修复版本:
- 2.1.14+
- 2.2.11+
- 2.3.5+
临时规避方法:
- 无
问题原因:
- brpc库兼容高版本内核的问题

trueeyu · 2022年11月28日 10:50

Not Found string in global dict

Unique表，查询报错

Not Found string in global dict

Github Issue: https://github.com/StarRocks/starrocks/issues/13262
Github Fix PR: https://github.com/StarRocks/starrocks/pull/13263
问题版本:
- 2.1.0 ~ 2.1.x
- 2.2.0 ~ 2.2.9
- 2.3.0 ~ 2.3.4
- 2.4.0
修复版本:
- 2.2.10+
- 2.3.5+
- 2.4.1+
临时规避方法:
- set global cbo_enable_low_cardinality_optimize=false;
问题原因:
- 低基数优化处理有问题

trueeyu · 2022年12月1日 03:23

build hash map crash

0x7f83e010a6d0(unknown)
0x256c340 starrocks::Agguegator::build_hash_map<>()
0x28ac22b starrocks::pipeline::Aggregatestreamingsinkoperator::_push_chunk_by_auto()
0x28b2635 starrocks::pipeline::Aggregatestreamingsinkoperator::push_chunk()
0x289716c starrocks::pipeline::PipelineDriver::process()
0x288c564 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
0x2132549 starrocks::ThreadPool::dispatch_thread()
0x212e0fa starrocks::Thread::supervise_thread()
0x7f83e0102e25 start_thread
0x7f83df71ebad__clone
0x0(unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/9360
Github Fix PR: https://github.com/StarRocks/starrocks/pull/9548
问题版本:
- 2.2.0 ~ 2.2.10
- 2.3.0
修复版本:
- 2.2.11+
- 2.3.1+
临时规避方法:
- 无
问题原因:
- 执行计划有问题

trueeyu · 2022年12月2日 08:21

Schema Change+Add Partition+Balance之后，内存中的元数据出错

*** SIGABRT (@0x3e900000b7a) received by PID 2938 (TID 0x7efe4114a700) from PID 2938; stack trace: ***
    @          0x33a1be2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7efe9dc36630 (unknown)
    @     0x7efe9cf6b3d7 __GI_raise
    @     0x7efe9cf6cac8 __GI_abort
    @          0x157860d _ZN9__gnu_cxx27__verbose_terminate_handlerEv.cold
    @          0x4d3dfe6 __cxxabiv1::__terminate()
    @          0x4d3e051 std::terminate()
    @          0x4d3e1a5 __cxa_throw
    @          0x157a1f1 std::__throw_length_error()
    @          0x16ef0d0 std::vector<>::_M_fill_insert()
    @          0x16fa218 starrocks::vectorized::NullableColumn::append_numbers()
    @          0x1abe001 starrocks::segment_v2::BitShufflePageDecoder<>::next_batch()
    @          0x1ab43b7 starrocks::segment_v2::ParsedPageV2::read()
    @          0x2cae998 starrocks::segment_v2::ScalarColumnIterator::next_batch()
    @          0x2c173e1 starrocks::vectorized::SegmentIterator::_do_get_next()
    @          0x2c1ace1 starrocks::vectorized::SegmentIterator::do_get_next()
    @          0x1b0a40a starrocks::SegmentIteratorWrapper::do_get_next()
    @          0x189116b starrocks::vectorized::TimedChunkIterator::do_get_next()
    @          0x190cf3d starrocks::vectorized::HeapMergeIterator::do_get_next()
    @          0x189116b starrocks::vectorized::TimedChunkIterator::do_get_next()
    @          0x19041f1 starrocks::vectorized::AggregateIterator::do_get_next()
    @          0x189116b starrocks::vectorized::TimedChunkIterator::do_get_next()
    @          0x18c701a starrocks::vectorized::TabletReader::do_get_next()
    @          0x1822aa7 starrocks::vectorized::Compaction::merge_rowsets()
    @          0x18249b2 starrocks::vectorized::Compaction::do_compaction_impl()
    @          0x182679b starrocks::vectorized::Compaction::do_compaction()
    @          0x1820904 starrocks::vectorized::CumulativeCompaction::compact()
    @          0x1745439 starrocks::StorageEngine::_perform_cumulative_compaction()
    @          0x17326cf starrocks::StorageEngine::_cumulative_compaction_thread_callback()
    @          0x4db83a0 execute_native_thread_routine
    @     0x7efe9dc2eea5 start_thread
    @     0x7efe9d0339fd __clone

Github Issue: https://github.com/StarRocks/starrocks/issues/5433
Github Fix PR: https://github.com/StarRocks/starrocks/pull/4541q
问题版本:
- 2.0.0 ~ 2.0.3
- 2.1.0 ~ 2.1.3
修复版本:
- 2.0.4+
- 2.1.4+
临时规避方法:
- 如果出现之后，需要删表重建。出问题需要满足三个原因。一般是升级会触发Balance，导致最终出现问题。建议升级之前关闭Balance, 升级完成之后开启Balance

admin set frontend config ("max_scheduling_tablets"="0");
admin set frontend config ("disable_balance"="true");
admin set frontend config ("disable_colocate_balance"="true");

admin set frontend config ("max_scheduling_tablets"="2000");
admin set frontend config ("disable_balance"="false");
admin set frontend config ("disable_colocate_balance"="false");

问题原因:
- 存储层复用SchemaID，在上述操作之下出现错乱。

trueeyu · 2022年12月3日 02:24

trueeyu · 2023年06月1日 03:58

OlapScanNode use-after-free

*** Aborted at 1666617333 (unix time) try "date -d @1666617333" if you are using GNU date ***
PC: @          0x2b5243b starrocks::ExprContext::close()
*** SIGSEGV (@0x60) received by PID 302338 (TID 0x7f58702a0700) from PID 96; stack trace: ***
    @          0x3cf55d2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f5921984630 (unknown)
    @          0x2b5243b starrocks::ExprContext::close()
    @          0x2b535c0 starrocks::Expr::close()
    @          0x27467e6 starrocks::vectorized::TabletScanner::close()
    @          0x2746e78 starrocks::vectorized::TabletScanner::~TabletScanner()
    @          0x246bbd7 _ZZN9starrocks10ObjectPool3addINS_10vectorized13TabletScannerEEEPT_S5_ENUlPvE_4_FUNES6_
    @          0x246b3ff starrocks::vectorized::OlapScanNode::~OlapScanNode()
    @          0x246ba22 starrocks::vectorized::OlapScanNode::~OlapScanNode()
    @          0x1ed7787 std::_Sp_counted_ptr<>::_M_dispose()
    @          0x18e7fda std::_Sp_counted_base<>::_M_release()
    @          0x1ed3e42 starrocks::RuntimeState::~RuntimeState()
    @          0x1e65a22 starrocks::FragmentExecState::~FragmentExecState()
    @          0x1e6edab std::_Sp_counted_ptr<>::_M_dispose()
    @          0x18e7fda std::_Sp_counted_base<>::_M_release()
    @          0x1e66db5

或堆栈打不全

*** SIGSEGV (@0x0) received by PID 3104 (TID 0x7f51b752b700) from PID 0; stack trace: ***
    @          0x3ff7972 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f522347c630 (unknown)
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/11395
Github Fix PR: https://github.com/StarRocks/starrocks/pull/11396
问题版本:
- 2.1.0 ~ 2.1.13
- 2.2.0 ~ 2.2.8
- 2.3.0 ~ 2.3.3
- 2.4.0
修复版本:
- 2.1.14+
- 2.2.9+
- 2.3.4+
- 2.4.1+
临时规避方法:
- 无
问题原因:
- 析构顺序问题

trueeyu · 2022年12月9日 07:21

array_agg crash

*** Aborted at 1670508783 (unix time) try "date -d @1670508783" if you are using GNU date ***
PC: @          0x28f91b5 starrocks::vectorized::NullableAggregateFunctionUnary<>::update_batch_selectively()
*** SIGSEGV (@0xa0) received by PID 4440 (TID 0x7f9a6a737700) from PID 160; stack trace: ***
    @          0x3ca37d2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f9ad08a3630 (unknown)
    @          0x28f91b5 starrocks::vectorized::NullableAggregateFunctionUnary<>::update_batch_selectively()
    @          0x26c9cd9 starrocks::Aggregator::compute_batch_agg_states_with_selection()
    @          0x263df4f starrocks::pipeline::AggregateStreamingSinkOperator::_push_chunk_by_auto()
    @          0x26446cd starrocks::pipeline::AggregateStreamingSinkOperator::push_chunk()
    @          0x2627d5d starrocks::pipeline::PipelineDriver::process()
    @          0x261da11 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x1f5a169 starrocks::ThreadPool::dispatch_thread()
    @          0x1f55d1a starrocks::Thread::supervise_thread()
    @     0x7f9ad089bea5 start_thread
    @     0x7f9acfeb6b0d __clone
    @                0x0 (unknown)

Github Issue: https://github.com/StarRocks/starrocks/issues/12073
Github Fix PR: https://github.com/StarRocks/starrocks/pull/12074
问题版本:
- 2.2.0 ~ 2.2.7
- 2.3.0 ~ 2.3.4
修复版本:
- 2.2.8+
- 2.3.5+
临时规避方法:
- 不使用array_agg
问题原因:
- 见issue描述

常见 Crash / BUG / 优化 查询

Window_funnel crash

查询中有 Like Crash

主键模型 compaction crash

建表超时报错

Tablet 磁盘间迁移 Crash

Grouping Sets crash

Java UDF Open Crash

Analyze too many versions

RoaringBitmap Crash

save meta crash

Sort Crash

Hudi 外表部分字段查询出来为Null

主键模型表使用 persistent index 导致磁盘空间膨胀

Ubuntu启动BE报错

Not Found string in global dict

build hash map crash

Schema Change+Add Partition+Balance之后，内存中的元数据出错

OlapScanNode use-after-free

array_agg crash

常见 Crash / BUG / 优化查询