常见 Crash / BUG / 优化查询

trueeyu · 2024年02月27日 06:14

UDF mem_tracker crash

*** Aborted at 1686120661 (unix time) try "date -d @1686120661" if you are using GNU date ***
PC: @          0x2ce0b61 starrocks::MemTracker::consume()
*** SIGSEGV (@0x0) received by PID 117789 (TID 0x7f9cd0728700) from PID 0; stack trace: ***
    @          0x58f9dc2 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f9d147ae362 (unknown)
    @     0x7f9d147b28b9 JVM_handle_linux_signal
    @     0x7f9d147a5f78 (unknown)
    @     0x7f9d13ce1390 (unknown)
    @          0x2ce0b61 starrocks::MemTracker::consume()
    @          0x486d328 _ZNSt17_Function_handlerIFvvEZN9starrocks24call_function_in_pthreadEPNS1_12RuntimeStateERKSt8functionIFNS1_6StatusEvEEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x475fbe0 starrocks::PriorityThreadPool::work_thread()
    @          0x58b97e7 thread_proxy
    @     0x7f9d13cd76ba start_thread
    @     0x7f9d132f841d clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/41710
Jira
问题版本:
- 2.5.0 ~ 2.5.19
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.8
- 3.2.0 ~ 3.2.3
修复版本:
- 2.5.20+
- 3.0.10+
- 3.1.9+
- 3.2.4+
问题原因:
临时解决办法:

trueeyu · 2024年03月11日 09:46

低基数导致查询报错

Dict Decode failed, Dict can't take cover all key :-424101658

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/40374
Jira
问题版本:
- 3.1.0 ~ 3.1.7
- 3.2.0 ~ 3.2.2
修复版本:
- 3.1.8+
- 3.2.3+
问题原因:
临时解决办法:
- set global cbo_enable_low_cardinality_optimize = false;

trueeyu · 2024年03月11日 09:46

UnpackConstColumnBinaryFunction crash

*** Aborted at 1709171849 (unix time) try "date -d @1709171849" if you are using GNU date ***
PC: @          0x3b4efa9 starrocks::UnpackConstColumnBinaryFunction<>::evaluate<>()
*** SIGSEGV (@0x7f21cb212000) received by PID 3809085 (TID 0x7f284d7ef700) from PID 18446744072822530048; stack trace: ***
    @          0x62f3702 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f29dfb8e240 (unknown)
    @          0x3b4efa9 starrocks::UnpackConstColumnBinaryFunction<>::evaluate<>()
    @          0x3b500cc starrocks::VectorizedBinaryPredicate<>::evaluate_checked()
    @          0x3a0c283 starrocks::ExprContext::evaluate()
    @          0x3a0c5cf starrocks::ExprContext::evaluate()
    @          0x2aaadb2 starrocks::ExecNode::eval_conjuncts()
    @          0x2a973ac starrocks::pipeline::Operator::eval_conjuncts_and_in_filters()
    @          0x2d6c9a6 starrocks::pipeline::SelectOperator::push_chunk()
    @          0x2a6996b starrocks::pipeline::PipelineDriver::process()
    @          0x5607bbe starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x4f07072 starrocks::ThreadPool::dispatch_thread()
    @          0x4f01b6a starrocks::Thread::supervise_thread()
    @     0x7f29dfb83f2b (unknown)
    @     0x7f29df9286bf clone
    @                0x0 (unknown)

或

** Aborted at 1709285311 (unix time) try "date -d @1709285311" if you are using GNU date ***
PC: @          0x585822c starrocks::StringFunctions::replace()
*** SIGSEGV (@0x268377d514b6) received by PID 23198 (TID 0x7fe017715700) from PID 2010453174; stack trace: ***
    @          0x6457542 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fe227651630 (/usr/lib64/libpthread-2.17.so;65d5e4de (deleted)+0xf62f)
    @          0x585822c starrocks::StringFunctions::replace()
    @          0x438b554 starrocks::VectorizedFunctionCallExpr::evaluate_checked()
    @          0x43222c9 starrocks::DictMappingExpr::evaluate_checked()
    @          0x43c587e starrocks::VectorizedInConstPredicate<>::evaluate_with_filter()
    @          0x43c5c25 starrocks::VectorizedInConstPredicate<>::evaluate_checked()
    @          0x4386389 starrocks::VectorizedIfExpr<>::evaluate_checked()
    @          0x3aca163 starrocks::ExprContext::evaluate()
    @          0x442146c starrocks::RuntimeFilterProbeCollector::update_selectivity()
    @          0x442259f starrocks::RuntimeFilterProbeCollector::evaluate()
    @          0x2b2fb16 starrocks::pipeline::Operator::eval_runtime_bloom_filters()
    @          0x2b35c40 starrocks::pipeline::ScanOperator::pull_chunk()
    @          0x2b01b47 starrocks::pipeline::PipelineDriver::process()
    @          0x5740e1e starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x500c91a starrocks::ThreadPool::dispatch_thread()
    @          0x50073aa starrocks::Thread::supervise_thread()
    @     0x7fe227649ea5 (/usr/lib64/libpthread-2.17.so;65d5e4de (deleted)+0x7ea4)
    @     0x7fe226a4ab0d (/usr/lib64/libc-2.17.so;65d5e4de (deleted)+0xfeb0c)
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/41740
Jira
问题版本:
- 2.5.0 ~ 2.5.19
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.8
- 3.2.0 ~ 3.2.3
修复版本:
- 2.5.20+
- 2.0.10+
- 3.1.9+
- 3.2.4+
问题原因:
临时解决办法:
- set global cbo_enable_low_cardinality_optimize = false;
- set global cbo_push_down_distinct_below_window = false;

trueeyu · 2024年03月5日 10:34

BE JVM 报错 Compressed class space

java.lang.OutOfMemoryError: Compressed class space

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/9944
Jira
问题版本:
- 2.2.0 ~ latest
- 2.3.0 ~ latest
- 2.4.0 ~ latest
修复版本:
- 2.5.0+
问题原因:
临时解决办法:
- 修改 be.conf 的 JAVA_OPTS 将 JVM 内存 限制调大

trueeyu · 2024年03月5日 12:33

array_distinct crash

*** Aborted at 1700019561 (unix time) try “date -d @1700019561” if you are using GNU date ***
PC: @ 0x4f69240 starrocks::vectorized::ArrayDistinct<>::_array_distinct_item<>()
*** SIGSEGV (@0x7f2288c00000) received by PID 27723 (TID 0x7f5880903700) from PID 18446744071708868608; stack trace: ***
@ 0x5aed0a2 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f59d4dc9630 (unknown)
@ 0x4f69240 starrocks::vectorized::ArrayDistinct<>::_array_distinct_item<>()
@ 0x4f69a9a starrocks::vectorized::ArrayDistinct<>::_array_distinct<>()
@ 0x4f69f24 starrocks::vectorized::ArrayFunctions::array_distinct_varchar()
@ 0x3fe0ab4 starrocks::vectorized::VectorizedFunctionCallExpr::evaluate_checked()
@ 0x38ea553 starrocks::ExprContext::evaluate()
@ 0x38ea89f starrocks::ExprContext::evaluate()
@ 0x3009ac4 starrocks::pipeline::ProjectOperator::push_chunk()
@ 0x2d6ab2e starrocks::pipeline::PipelineDriver::process()
@ 0x510747a starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x4af5bc2 starrocks::ThreadPool::dispatch_thread()
@ 0x4af06ba starrocks:
:supervise_thread()
@ 0x7f59d4dc1ea5 start_thread
@ 0x7f59d43dc96d __clone
@ 0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/36377
Jira
问题版本:
- 2.5.0 ~ 2.5.16
- 3.1.0 ~ 3.1.5
- 3.2.0
修复版本:
- 2.5.17+
- 3.1.6+
- 3.2.1+
问题原因:
临时解决办法:

trueeyu · 2024年05月23日 06:11

主键模型导入 crash

*** Aborted at 1709773243 (unix time) try “date -d @1709773243” if you are using GNU date ***
PC: @ 0x41fdae7 starrocks::ShardByLengthMutableIndex::memory_usage()
*** SIGSEGV (@0x0) received by PID 195386 (TID 0x7f4bc4020700) from PID 0; stack trace: ***
@ 0x5824342 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f535d7cd630 (unknown)
@ 0x41fdae7 starrocks::ShardByLengthMutableIndex::memory_usage()
@ 0x4069bd3 starrocks::UpdateManager::on_rowset_finished()
@ 0x467d357 starrocks::vectorized::DeltaWriter::commit()
@ 0x51019ab starrocks::vectorized::AsyncDeltaWriter::_execute()
@ 0x596ae2c bthread::ExecutionQueueBase::_execute()
@ 0x596bba8 bthread::ExecutionQueueBase::_execute_tasks()
@ 0x48c3b92 starrocks::ThreadPool::dispatch_thread()
@ 0x48be68a starrocks:
:supervise_thread()
@ 0x7f535d7c5ea5 start_thread
@ 0x7f535cde0b0d __clone
@ 0x0 (unknown)

或

*** Aborted at 1710326728 (unix time) try "date -d @1710326728" if you are using GNU date ***
PC: @          0x46f6714 starrocks::PrimaryIndex::memory_usage()
*** SIGSEGV (@0x88) received by PID 22619 (TID 0x7fb8b63f8700) from PID 136; stack trace: ***
    @          0x62e4042 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fb9481cb630 (unknown)
    @          0x46f6714 starrocks::PrimaryIndex::memory_usage()
    @          0x4810df6 starrocks::UpdateManager::on_rowset_finished()
    @          0x4e60687 starrocks::DeltaWriter::commit()
    @          0x49da1ce _ZNSt17_Function_handlerIFvvEZN9starrocks17SegmentFlushToken6submitEPN4brpc10ControllerEPKNS1_30PTabletWriterAddSegmentRequestEPNS1_29PTabletWriterAddSegmentResultEPN6google8protobuf7ClosureEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x50e93c2 starrocks::ThreadPool::dispatch_thread()
    @          0x50e3eba starrocks::Thread::supervise_thread()
    @     0x7fb9481c3ea5 start_thread
    @     0x7fb9477de96d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
Jira
问题版本:
- 2.5.0 ~ 2.5.20
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.10
- 3.2.0 ~ 3.2.6
修复版本:
- 2.5.21+
- 3.0.10+
- 3.1.11+
- 3.2.7+
问题原因:
临时解决办法:

trueeyu · 2024年05月20日 11:01

高频delete会导致be crash: version_for_delete_predicate

*** Aborted at 1710223298 (unix time) try “date -d @1710223298” if you are using GNU date ***
PC: @ 0x4591244 starrocks::TabletMeta::version_for_delete_predicate()
*** SIGSEGV (@0x2930000030a) received by PID 11091 (TID 0x7f0c54fff700) from PID 778; stack trace: ***
@ 0x63911c2 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f1b591e7630 (unknown)
@ 0x4591244 starrocks::TabletMeta::version_for_delete_predicate()
@ 0x4793b9a starrocks::SizeTieredCompactionPolicy::_pick_rowsets_to_size_tiered_compact()
@ 0x47968f0 starrocks::SizeTieredCompactionPolicy::need_compaction()
@ 0x4563e01 starrocks::Tablet::need_compaction()
@ 0x4788c3d starrocks::CompactionManager::update_tablet()
@ 0x4788f31 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager16_dispatch_workerEvEUlvE_E9_M_invokeERKSt9_Any_data
@ 0x4f836b2 starrocks::ThreadPool::dispatch_thread()
@ 0x4f7e14a starrocks::supervise_thread()
@ 0x7f1b591dfea5 start_thread
@ 0x7f1b585e0b0d __clone
@ 0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/42100
- https://github.com/StarRocks/starrocks/pull/42174
Jira
问题版本:
- 2.5.0 ~ 2.5.19
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.9
- 3.2.0 ~ 3.2.4
修复版本:
- 2.5.20+
- 2.0.10+
- 3.1.10+
- 3.2.5+
问题原因:
临时解决办法:
- be.conf enable_size_tiered_compaction_strategy=false

trueeyu · 2024年03月18日 09:35

Plan check 报错

Type check failed. the type of arg 65: sum in expr ‘sum(65: sum)’ is defined as INT, but the actual type is BIGINT
at com.starrocks.sql.optimizer.validate.PlanValidator.validatePlan(PlanValidator.java:64) ~[starrocks-fe.jar:?]
at com.starrocks.sql.optimizer.Optimizer.optimizeByCost(Optimizer.java:235) ~[starrocks-fe.jar:?]
at com.starrocks.sql.optimizer.Optimizer.optimize(Optimizer.java:136) ~[starrocks-fe.jar:?]
at com.starrocks.sql.StatementPlanner.createQueryPlan(StatementPlanner.java:151) ~[starrocks-fe.jar:?]
at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:99) ~[starrocks-fe.jar:?]
at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:62) ~[starrocks-fe.jar:?]
at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:482) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:375) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:481) ~[starrocks-fe.jar:?]
at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:767) ~[starrocks-fe.jar:?]
at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69) ~[starrocks-fe.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/42454
Jira
问题版本:
- 3.1.0 ~ 3.1.9
- 3.2.0 ~ 3.2.4
修复版本:
- 3.1.10+
- 3.2.5+
问题原因:
临时解决办法:

trueeyu · 2024年03月18日 09:42

BE TabletSchemaMap 死锁

Thread 1331 (Thread 0x7f697b1fe700 (LWP 98875)):
#0  0x00007fe5a12794ed in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007fe5a1274dcb in _L_lock_883 () from /lib64/libpthread.so.0
#2  0x00007fe5a1274c98 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000042f5efb in starrocks::TabletSchemaMap::emplace(starrocks::TabletSchemaPB const&) ()
#4  0x00000000042d673d in starrocks::TabletMeta::init_from_pb(starrocks::TabletMetaPB*) ()
#5  0x00000000042b4e42 in starrocks::Tablet::generate_tablet_meta_copy_unlocked(std::shared_ptr<starrocks::TabletMeta> const&) const ()
#6  0x000000000429d05e in starrocks::SnapshotManager::snapshot_full[abi:cxx11](std::shared_ptr<starrocks::Tablet> const&, long, long, bool) ()
#7  0x000000000429f223 in starrocks::SnapshotManager::make_snapshot(starrocks::TSnapshotRequest const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) ()
#8  0x0000000002caccd1 in starrocks::AgentServer::Impl::make_snapshot(starrocks::TAgentResult&, starrocks::TSnapshotRequest const&) ()
#9  0x0000000004c4f94d in starrocks::BackendServiceProcessor::process_make_snapshot(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) ()
#10 0x0000000004c55c92 in starrocks::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, void*) ()
#11 0x0000000004c579a2 in apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*) ()
#12 0x0000000005c081b8 in apache::thrift::server::TConnectedClient::run() ()
#13 0x0000000005c006b4 in apache::thrift::server::TThreadedServer::TConnectedClientRunner::run() ()
#14 0x0000000005c02ebd in apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>) ()
#15 0x0000000005be8626 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread> > > >::_M_run() ()
#16 0x0000000008133fa0 in execute_native_thread_routine ()
#17 0x00007fe5a1272dd5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fe5a088dead in clone () from /lib64/libc.so.6

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/40406
Jira
问题版本:
- 2.5.0 ～ 2.5.18
- 3.0.0 ～ 3.0.9
- 3.1.0 ～ 3.1.8
- 3.2.0 ～ 3.2.2
修复版本:
- 2.5.19+
- 3.0.10+
- 3.1.9+
- 3.2.3+
问题原因:
临时解决办法:

trueeyu · 2024年03月22日 11:15

Compaciton 大文件 crash

   @          0x5e50559 google::LogMessageFatal::~LogMessageFatal()
    @          0x27ecfaf starrocks::BinaryColumnBase<>::_build_slices()
    @          0x2757dd5 starrocks::BinaryColumnBase<>::raw_data()
    @          0x480a0da starrocks::ScalarColumnWriter::append()
    @          0x480a33c starrocks::StringColumnWriter::append()
    @          0x43a2806 starrocks::SegmentWriter::append_chunk()
    @          0x4a98da0 starrocks::VerticalRowsetWriter::add_columns()
    @          0x4a73ac7 starrocks::VerticalCompactionTask::_compact_data()
    @          0x4a74d0f starrocks::VerticalCompactionTask::_compact_column_group()
    @          0x4a75632 starrocks::VerticalCompactionTask::_vertical_compaction_data()
    @          0x4a76129 starrocks::VerticalCompactionTask::run_impl()
    @          0x4a6b6fc starrocks::CompactionTask::run()
    @          0x44d30d3 _ZNSt17_Function_handlerIFvvEZN9starrocks17CompactionManager9_scheduleEvEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x4cc33d2 starrocks::ThreadPool::dispatch_thread()
    @          0x4cbde6a starrocks::Thread::supervise_thread()
    @     0x7f4658c6644b start_thread
    @     0x7f465803f40f __GI___clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/36281
Jira
问题版本:
- 2.5.0 ~ 2.5.20
- 3.0.0 ~ 3.0.8
- 3.1.0 ~ 3.1.5
- 3.2.0
修复版本:
- 2.5.21+
- 3.0.9+
- 3.1.7+
- 3.2.1+
问题原因:
临时解决办法:

trueeyu · 2024年04月23日 14:50

Persistent index 导致 BE 启动失败

*** Aborted at 1711349618 (unix time) try "date -d @1711349618" if you are using GNU date ***
PC: @     0x7f39cefe5387 __GI_raise
*** SIGABRT (@0x3e80000c50f) received by PID 50447 (TID 0x7f39425ff700) from PID 50447; stack trace: ***
    @          0x5b1ba42 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f39cfa9a630 (unknown)
    @     0x7f39cefe5387 __GI_raise
    @     0x7f39cefe6a78 __GI_abort
    @          0x2cdf2be starrocks::failure_function()
    @          0x5b0f41d google::LogMessage::Fail()
    @          0x5b1188f google::LogMessage::SendToLog()
    @          0x5b0ef6e google::LogMessage::Flush()
    @          0x5b11e99 google::LogMessageFatal::~LogMessageFatal()
    @          0x4265eaf starrocks::TabletUpdates::_apply_rowset_commit()
    @          0x4266353 starrocks::TabletUpdates::do_apply()
    @          0x4b17465 starrocks::ThreadPool::dispatch_thread()
    @          0x4b11e4a starrocks::Thread::supervise_thread()
    @     0x7f39cfa92ea5 start_thread
    @     0x7f39cf0ad96d __clone
    @                0x0 (unknown)

W0325 14:58:06.988127 52911 rowset.cpp:141] Fail to open /data/starrocks/storage/be/data/1021/321209132/1248468755/0200000000005e48a648e0f47d7f27a81aa03e6bcc4b45b4_0.dat: Corruption: Bad segment file /data/starrocks/storage/be/data/1021/321209132/1248468755/0200000000005e48a648e0f47d7f27a81aa03e6bcc4b45b4_0.dat: file size 0 < 12
/build/starrocks/be/src/storage/rowset/segment.cpp:195 Segment::parse_segment_footer(read_file.get(), &footer, footer_length_hint, partial_rowset_footer)
/build/starrocks/be/src/storage/rowset/segment.cpp:67 segment->_open(footer_length_hint, partial_rowset_footer)
W0325 14:58:06.988687 52911 rowset_update_state.cpp:39] load RowsetUpdateState error: Corruption: Bad segment file /data/starrocks/storage/be/data/1021/321209132/1248468755/0200000000005e48a648e0f47d7f27a81aa03e6bcc4b45b4_0.dat: file size 0 < 12
/build/starrocks/be/src/storage/rowset/segment.cpp:195 Segment::parse_segment_footer(read_file.get(), &footer, footer_length_hint, partial_rowset_footer)
/build/starrocks/be/src/storage/rowset/segment.cpp:67 segment->_open(footer_length_hint, partial_rowset_footer)
/build/starrocks/be/src/storage/rowset/rowset.cpp:75 do_load()
/build/starrocks/be/src/storage/rowset/rowset.cpp:454 load()
/build/starrocks/be/src/storage/rowset_update_state.cpp:161 _load_upserts(rowset, 0, pk_column.get()) tablet:321209132 stack:
    @          0x46a4e19  _ZZSt9call_onceIZN9starrocks17RowsetUpdateState4loadEPNS0_6TabletEPNS0_6RowsetEEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv
    @     0x7fb3c0d7b20b  __pthread_once_slow
    @          0x469fde7  starrocks::RowsetUpdateState::load()
    @          0x4262183  starrocks::TabletUpdates::_apply_rowset_commit()
    @          0x4266353  starrocks::TabletUpdates::do_apply()
    @          0x4b17465  starrocks::ThreadPool::dispatch_thread()
    @          0x4b11e4a  starrocks::Thread::supervise_thread()
    @     0x7fb3c0d7cea5  start_thread
    @     0x7fb3c039796d  __clone
    @              (nil)  (unknown)

get_applied_rowsets failed, tablet updates is in error state: tablet:85018 actual row size changed after compaction 50531 -> 50041tablet:85018 #version:13 [29445 29456.1@12 29456.1] #pending:0 backend

F0423 22:31:29.743636 475679 tablet_updates.cpp:1132] delvec inconsistent tablet:8858730 rssid:5262 #old:1402 #add:4 #new:1402 old_v:10497 v:10498

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/30841
Jira
问题版本:
- 2.5.0 ~ 2.5.20
- 3.0.0 ~ 3.0.8
- 3.1.0 ~ 3.1.5
修复版本:
- 2.5.21+
- 3.0.9+
- 3.1.6+
问题原因:
临时解决办法:
- 使用 ./meta_tool.sh --operation=delete_persistent_index_meta 功能删除有问题 tablet 的 persistent index 并重新启动。如果是3副本的话可以使用 ./meta_tool.sh --operation=delete_meta功能删除

trueeyu · 2024年04月9日 10:34

trueeyu · 2024年04月10日 03:09

BE manual_compact 线程占用大量磁盘IO

Github Issue:
Github Fix PR:
Jira
问题版本:
- 2.5.0 ～ 2.5.20
- 3.0.0 ～ 3.0.9
- 3.1.0 ～ 3.1.9
修复版本:
- 2.5.21+
- 3.0.10+
- 3.1.10+
问题原因:
临时解决办法:
- 修改be.conf update_compaction_size_threshold=67108864

trueeyu · 2024年04月10日 05:36

查询 _finish_late_materialization crash

*** Aborted at 1699870338 (unix time) try "date -d @1699870338" if you are using GNU date ***
PC: @          0x435387b starrocks::vectorized::SegmentIterator::_finish_late_materialization()
*** SIGSEGV (@0xd0) received by PID 25789 (TID 0x7fd423b90700) from PID 208; stack trace: ***
    @          0x5b97b22 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fd6797af630 (unknown)
    @          0x435387b starrocks::vectorized::SegmentIterator::_finish_late_materialization()
    @          0x435c790 starrocks::vectorized::SegmentIterator::_do_get_next()
    @          0x435f270 starrocks::vectorized::SegmentIterator::do_get_next()
    @          0x43e3f42 starrocks::vectorized::ProjectionIterator::do_get_next()
    @          0x4994e95 starrocks::SegmentIteratorWrapper::do_get_next()
    @          0x47c41c3 starrocks::vectorized::TimedChunkIterator::do_get_next()
    @          0x4415466 starrocks::vectorized::TabletReader::do_get_next()
    @          0x304a28b starrocks::pipeline::OlapChunkSource::_read_chunk_from_storage()
    @          0x304a976 starrocks::pipeline::OlapChunkSource::_read_chunk()
    @          0x303a22f starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking()
    @          0x2db3f14 _ZZN9starrocks8pipeline12ScanOperator18_trigger_next_scanEPNS_12RuntimeStateEiENKUlvE_clEv
    @          0x2dc533e starrocks::workgroup::ScanExecutor::worker_thread()
    @          0x4b968f2 starrocks::ThreadPool::dispatch_thread()
    @          0x4b9138a starrocks::Thread::supervise_thread()
    @     0x7fd6797a7ea5 start_thread
    @     0x7fd678dc2b0d __clone
    @                0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/42994
Jira
问题版本:
- 2.5.0 ~ 2.5.20
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.9
- 3.2.0 ~ 3.2.4
修复版本:
- 2.5.21+
- 3.0.10+
- 3.1.10+
- 3.2.5+
问题原因:
临时解决办法:
- set global enable_filter_unused_columns_in_scan_stage = false

trueeyu · 2024年04月19日 08:29

Join cancel crash

*** SIGSEGV (@0x0) received by PID 555045 (TID 0x7f944a3ad700) from PID 0; stack trace: ***
@ 0x481e332 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f950f8a73ab os::Linux::chained_handler()
@ 0x7f950f8abefc JVM_handle_linux_signal
@ 0x7f950f89ed48 signalHandler()
@ 0x7f950ef72420 (unknown)
@ 0x2f35d3f starrocks::vectorized::HashJoiner::_has_null()
@ 0x2ec31ea starrocks::pipeline::HashJoinBuildOperator::set_finishing()
@ 0x2e76ff7 starrocks::pipeline::PipelineDriver::_mark_operator_finishing()
@ 0x2e77099 starrocks::pipeline::PipelineDriver::_mark_operator_finished()
@ 0x2e77629 starrocks::pipeline::PipelineDriver::_mark_operator_cancelled()
@ 0x2e779b2 starrocks::pipeline::PipelineDriver::_check_fragment_is_canceled()
@ 0x2e77dd0 starrocks::pipeline::PipelineDriver::process()
@ 0x2e6e5a3 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x2680a05 starrocks::ThreadPool::dispatch_thread()
@ 0x267bf2a starrocks::supervise_thread()
@ 0x7f950ef66609 start_thread
@ 0x7f950ed2c133 clone
@ 0x0 (unknown)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/32219
Jira
问题版本:
- 2.3.0 ~ 2.3.18
- 2.4.0 ~ latest
- 2.5.0 ~ 2.5.16
- 3.0.0 ~ 3.0.6
- 3.1.0 ~ 3.1.3
修复版本:
- 2.3.19+
- 2.4 没有 fix
- 2.5.17+
- 3.0.7+
- 3.1.4+
问题原因:
临时解决办法:

trueeyu · 2024年05月22日 02:23

Schema change 后，unique 模型/agg模型的表查询结果不对

有时候会有重复数据

一般添加删除索引等SchemaChange后，查询 where 条件中有 key 列，容易触发

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/44854
Jira
问题版本:
- 2.5.0 ～ 2.5.20
- 3.0.0 ～ 3.0.9
- 3.1.0 ～ 3.1.11
- 3.2.0 ～ 3.2.6
修复版本:
- 2.5.21
- 3.0.10
- 3.1.12
- 3.2.7
问题原因:
解决办法:
- 需要升级后，并且重建表才能解决。

trueeyu · 2024年05月8日 10:02

Unique表或是Agg表，order by Desc limit 查询结果不对

Order by xxx desc limit

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/45037
- https://github.com/StarRocks/starrocks/pull/45285
Jira
问题版本:
- 3.0.0 ~ 3.0.9
- 3.1.0 ~ 3.1.11
- 3.2.0 ~ 3.2.6
修复版本:
- 3.0.10+
- 3.1.12+
- 3.2.7+
问题原因:
解决办法:

trueeyu · 2024年05月7日 12:08

FE CPU打满，大量查询超时

Jstack 有如下堆栈

"starrocks-mysql-nio-pool-2791" #119533 daemon prio=5 os_prio=0 tid=0x00007fe65c060800 nid=0x19244 runnable [0x00007fe629881000]
   java.lang.Thread.State: RUNNABLE
        at java.util.Arrays.hashCode(Arrays.java:4146)
        at java.util.Objects.hash(Objects.java:128)
        at com.starrocks.sql.optimizer.base.DistributionCol.hashCode(DistributionCol.java:116)
        at java.util.HashMap.hash(HashMap.java:340)
        at java.util.HashMap.get(HashMap.java:558)
        at com.starrocks.sql.optimizer.base.DistributionDisjointSet.find(DistributionDisjointSet.java:59)
        at com.starrocks.sql.optimizer.base.DistributionDisjointSet.union(DistributionDisjointSet.java:73)
        at com.starrocks.sql.optimizer.base.DistributionSpec$PropertyInfo.unionNullRelaxCols(DistributionSpec.java:98)
        at com.starrocks.sql.optimizer.OutputPropertyDeriver.computeHashJoinDistributionPropertyInfo(OutputPropertyDeriver.java:183)
        at com.starrocks.sql.optimizer.OutputPropertyDeriver.visitPhysicalJoin(OutputPropertyDeriver.java:259)
        at com.starrocks.sql.optimizer.OutputPropertyDeriver.visitPhysicalHashJoin(OutputPropertyDeriver.java:199)
        at com.starrocks.sql.optimizer.OutputPropertyDeriver.visitPhysicalHashJoin(OutputPropertyDeriver.java:76)
        at com.starrocks.sql.optimizer.operator.physical.PhysicalHashJoinOperator.accept(PhysicalHashJoinOperator.java:41)
        at com.starrocks.sql.optimizer.OutputPropertyDeriver.getOutputProperty(OutputPropertyDeriver.java:95)
        at com.starrocks.sql.optimizer.task.EnforceAndCostTask.execute(EnforceAndCostTask.java:206)
        at com.starrocks.sql.optimizer.task.SeriallyTaskScheduler.executeTasks(SeriallyTaskScheduler.java:69)
        at com.starrocks.sql.optimizer.Optimizer.memoOptimize(Optimizer.java:571)
        at com.starrocks.sql.optimizer.Optimizer.optimizeByCost(Optimizer.java:188)
        at com.starrocks.sql.optimizer.Optimizer.optimize(Optimizer.java:126)
        at com.starrocks.sql.StatementPlanner.createQueryPlanWithReTry(StatementPlanner.java:203)
        at com.starrocks.sql.StatementPlanner.planQuery(StatementPlanner.java:119)
        at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:88)
        at com.starrocks.sql.StatementPlanner.plan(StatementPlanner.java:57)
        at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:436)
        at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:362)
        at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:476)
        at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:742)
        at com.starrocks.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:69)
        at com.starrocks.mysql.nio.ReadListener$Lambda$737/1304093818.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

   Locked ownable synchronizers:
        - <0x00000004efd52070> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/34140
Jira
问题版本:
- 3.1.0 ~ 3.1.4
修复版本:
- 3.1.5+
问题原因:
解决办法:

trueeyu · 2024年05月7日 12:17

3.2 版本，有非等值 on 条件的 join 结果不对

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/45117
Jira
问题版本:
- 3.2.0 ~ 3.2.6
修复版本:
- 3.2.7+
问题原因:
临时解决办法:
- set global set interleaving_group_size=0

trueeyu · 2024年08月2日 02:18

The tablet write operation update metadata take a long time

The tablet write operation update metadata take a long time

Github Issue:
Github Fix PR:
- https://github.com/StarRocks/starrocks/pull/40725
- https://github.com/StarRocks/starrocks/pull/46733
Jira
问题版本:
- 2.5.0 ~ 2.5.21
- 3.0.0 ~ latest
- 3.1.0 ~ 3.1.12
- 3.2.0 ~ 3.2.8
修复版本:
- 2.5.22+
- 3.0 未 fix
- 3.1.13+
- 3.2.9+
问题原因:
临时解决办法:

常见 Crash / BUG / 优化 查询

UDF mem_tracker crash

低基数导致查询报错

UnpackConstColumnBinaryFunction crash

BE JVM 报错 Compressed class space

array_distinct crash

主键模型导入 crash

高频delete会导致be crash: version_for_delete_predicate

Plan check 报错

BE TabletSchemaMap 死锁

Compaciton 大文件 crash

Persistent index 导致 BE 启动失败

BE manual_compact 线程占用大量磁盘IO

查询 _finish_late_materialization crash

Join cancel crash

Schema change 后，unique 模型/agg模型的表查询结果不对

Unique表或是Agg表，order by Desc limit 查询结果不对

FE CPU打满，大量查询超时

3.2 版本，有非等值 on 条件的 join 结果不对

The tablet write operation update metadata take a long time

常见 Crash / BUG / 优化查询