常见 Crash / BUG / 优化 查询

  1. Load 内存持续增长不释放

  1. 主键模型 load_delete crash

*** SIGABRT (@0x8a5c) received by PID 35420 (TID 0x7f93fa5f7700) from PID 35420; stack trace: ***
    @          0x77e7a52 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f96429345f0 (unknown)
    @     0x7f964258d337 __GI_raise
    @     0x7f964258ea28 __GI_abort
    @          0x34ed0ee starrocks::failure_function()
    @          0x77db42d google::LogMessage::Fail()
    @          0x77dd89f google::LogMessage::SendToLog()
    @          0x77daf7e google::LogMessage::Flush()
    @          0x77ddea9 google::LogMessageFatal::~LogMessageFatal()
    @          0x79ba804 brpc::BaiduStreamingLogHandler()
    @          0x7b4068a google::protobuf::internal::LogMessage::Finish()
    @          0x7b7eaf0 google::protobuf::RepeatedPtrField<>::Get()
    @          0x61349f6 starrocks::lake::RowsetUpdateState::load_delete()
    @          0x6124377 starrocks::lake::UpdateManager::publish_primary_key_tablet()
    @          0x6163c61 starrocks::lake::PrimaryKeyTxnLogApplier::apply_write_log()
  1. Nest-Loop-Join crash

*** Aborted at 1724088138 (unix time) try "date -d @1724088138" if you are using GNU date ***
PC: @     0x7f059e043895 __memmove_avx_unaligned_erms
*** SIGSEGV (@0x0) received by PID 1351 (TID 0x7f0472519700) from PID 0; stack trace: ***
    @          0x67ec202 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f059ec5fc20 (unknown)
    @     0x7f059e043895 __memmove_avx_unaligned_erms
    @          0x2c3f594 std::vector<>::_M_range_insert<>()
    @          0x2c43014 starrocks::BinaryColumnBase<>::append()
    @          0x347fe0e starrocks::NullableColumn::append()
    @          0x3464ba2 starrocks::Chunk::append()
    @          0x51e4000 starrocks::ChunkAccumulator::push()
    @          0x38331f1 starrocks::pipeline::NLJoinBuildChunkStreamBuilder::init()
    @          0x38334a7 starrocks::pipeline::NLJoinContext::finish_one_right_sinker()
    @          0x383bc1a starrocks::pipeline::NLJoinBuildOperator::set_finishing()
    @          0x3889f8b starrocks::pipeline::PipelineDriver::_mark_operator_finishing()
    @          0x388a099 starrocks::pipeline::PipelineDriver::_mark_operator_finished()
    @          0x388a69b starrocks::pipeline::PipelineDriver::_mark_operator_cancelled()
    @          0x388acba starrocks::pipeline::PipelineDriver::cancel_operators()
    @          0x387dc3b starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x2e896ac starrocks::ThreadPool::dispatch_thread()
    @          0x2e8332a starrocks::Thread::supervise_thread()
    @     0x7f059ec5517a start_thread
    @     0x7f059dfdfdc3 __GI___clone
    @                0x0 (unknown)
  1. 窗口函数 crash

*** Aborted at 1724033491 (unix time) try "date -d @1724033491" if you are using GNU date ***
PC: @          0x2e22a51 starrocks::Analytor::current_chunk_size()
*** SIGSEGV (@0x0) received by PID 37179 (TID 0x2b2444db8700) from PID 0; stack trace: ***
    @          0x5c30f02 google::(anonymous namespace)::FailureSignalHandler()
    @     0x2b23aa3aa630 (unknown)
    @          0x2e22a51 starrocks::Analytor::current_chunk_size()
    @          0x30e3b0f starrocks::pipeline::AnalyticSinkOperator::_process_by_partition_if_necessary_for_unbounded_preceding_range_frame_streaming()
    @          0x30e3209 starrocks::pipeline::AnalyticSinkOperator::set_finishing()
    @          0x2db6969 starrocks::pipeline::PipelineDriver::_mark_operator_finishing()
    @          0x2db6a19 starrocks::pipeline::PipelineDriver::_mark_operator_finished()
    @          0x2db700b starrocks::pipeline::PipelineDriver::_mark_operator_cancelled()
    @          0x2db75fa starrocks::pipeline::PipelineDriver::cancel_operators()
    @          0x5206ef6 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x4be8fe2 starrocks::ThreadPool::dispatch_thread()
    @          0x4be3a7a starrocks::Thread::supervise_thread()
    @     0x2b23aa3a2ea5 start_thread
    @     0x2b23aadc3b0d __clone
    @                0x0 (unknown)

248. Invalid TType

W0805 16:55:14.940848  1227 thrift_rpc_helper.cpp:86] call frontend service failed, address=TNetworkAddress(=****/), port=9020), reason=invalid TType
  1. log_error 死锁,导致频繁打 cancel 日志,CPU高

query_pool 内存不释放

5 tids: 814,817,819,825,833
         0x7e6ea98  starrocks::RuntimeState::log_error(starrocks::Status const&)
         0x7eaf680  starrocks::pipeline::ExchangeSinkOperator::set_finishing(starrocks::RuntimeState*)
         0x550130c  starrocks::pipeline::PipelineDriver::_mark_operator_finishing(std::shared_ptr<starrocks::pipeline::Operator>&, starrocks::RuntimeState*)
         0x5502ce8  starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int)
         0x7e4e6e8  starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
         0x81649ec  starrocks::ThreadPool::dispatch_thread()
         0x815dc5c  starrocks::Thread::supervise_thread(void*)
    0xffff838fd5c8  (/usr/lib/aarch64-linux-gnu/libc.so.6+0x7d5c8)
    0xffff83965edc  (/usr/lib/aarch64-linux-gnu/libc.so.6+0xe5edc)
  1. Having 未处理导致查询结果不对

  1. 升级后 broker load 性能下降

  1. Spill crash

@ 0xffffa79b25d4 (/usr/lib/aarch64-linux-gnu/libc.so.6+0x825d3)
@ 0x94841f8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
@ 0xffffa8e24850 ([vdso]+0x84f)
@ 0x55802ec starrocks::spill::LogBlockContainer::append_data(std::vector<starrocks::Slice, std::allocatorstarrocks::Slice > const&, unsigned long)
@ 0x5585554 starrocks::spill::LogBlock::append(std::vector<starrocks::Slice, std::allocatorstarrocks::Slice > const&)
@ 0x56f5938 starrocks::spill::BlockSpillOutputDataStream::append(starrocks::RuntimeState*, std::vector<starrocks::Slice, std::allocatorstarrocks::Slice > const&, unsigned long, unsigned long)
@ 0x56ebc0c starrocks::spill::ColumnarSerde::serialize(starrocks::RuntimeState*, starrocks::spill::SerdeContext&, std::shared_ptrstarrocks::Chunk const&, std::shared_ptrstarrocks::spill::SpillOutputDataStream const&, bool)
@ 0x56c6dc8 starrocks::spill::UnorderedMemTable::finalize(starrocks::workgroup::YieldContext&, std::shared_ptrstarrocks::spill::SpillOutputDataStream const&)
@ 0x56bd814 starrocks::spill::PartitionedSpillerWriter::spill_partition(starrocks::workgroup::YieldContext&, starrocks::spill::SerdeContext&, starrocks::spill::SpilledPartition*)
  1. 频繁触发低基数收集报错

偶尔出现这种日志是正常的,频繁出现是不正常的

WARN (ForkJoinPool.commonPool-worker-5|209) [DefaultCoordinator.getNext():786] query failed:     global dict greater than DICT_DECODE_MAX_SIZE
  1. CN 使用华为 obs crash

F0715 03:28:20.325001 1906116 vacuum.cpp:555] Duplicate files were returned from the remote storage. The most likely cause is an S3 or HDFS API compatibility issue with your remote storage implementation. duplicate file: staros://1969195/meta/00000000001E0C2B_000000000000118D.meta
*** Check failure stack trace: ***

*** Aborted at 1720985300 (unix time) try "date -d @1720985300" if you are using GNU date ***
PC: @     0x7f82783e124f (unknown)
*** SIGABRT (@0x7d000220f0e) received by PID 2232078 (TID 0x7f7e66dfe640) from PID 2232078; stack trace: ***
    @          0x653d562 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f82783951f0 (unknown)
    @     0x7f82783e124f (unknown)
    @     0x7f8278395146 raise
    @     0x7f82783804f7 abort
    @          0x2a6487e starrocks::failure_function()
    @          0x6530f3d google::LogMessage::Fail()
    @          0x65333af google::LogMessage::SendToLog()
    @          0x6530a8e google::LogMessage::Flush()
    @          0x65339b9 google::LogMessageFatal::~LogMessageFatal()
    @          0x50a3439 _ZNSt17_Function_handlerIFbSt17basic_string_viewIcSt11char_traitsIcEEEZN9starrocks4lake19delete_tablets_implEPNS6_13TabletManagerERKNSt7__cxx1112basic_stringIcS2_SaIcEEERKSt6vectorIlSaIlEEEUlS3_E0_E9_M_invokeERKSt9_Any_dataOS3_
    @          0x4cf5276 _ZNSt17_Function_handlerIFbN6staros7starlet5fslib9EntryStatEEZN9starrocks17StarletFileSystem11iterate_dirERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt8functionIFbSt17basic_string_viewIcSA_EEEEUlS3_E_E9_M_invokeERKSt9_Any_dataOS3_
    @          0x5cec1c5 _ZN6staros7starlet5fslib12S3FileSystem8list_dirESt17basic_string_viewIcSt11char_traitsIcEES6_S6_bSt10shared_ptrIN3Aws2S38S3ClientEESt8functionIFbNS1_9EntryStatEEE.localalias
    @          0x5cec713 staros::starlet::fslib::S3FileSystem::list_dir()
    @          0x5cfdfcf staros::starlet::fslib::CacheFileSystemImpl::list_dir()
    @          0x5cfc765 staros::starlet::fslib::CacheFileSystem::list_dir()
    @          0x4cf9ea2 starrocks::StarletFileSystem::iterate_dir()
    @          0x509fbbf starrocks::lake::delete_tablets_impl()
    @          0x50a1227 starrocks::lake::delete_tablets()
    @          0x2c653ce _ZNSt17_Function_handlerIFvvEZN9starrocks15LakeServiceImpl13delete_tabletEPN6google8protobuf13RpcControllerEPKNS1_4lake19DeleteTabletRequestEPNS7_20DeleteTabletResponseEPNS4_7ClosureEEUlvE_E9_M_invokeERKSt9_Any_data
    @          0x2d0d64c starrocks::ThreadPool::dispatch_thread()
    @          0x2d072fa starrocks::Thread::supervise_thread()
    @     0x7f82783df67a (unknown)
    @     0x7f82784625e0 (unknown)
    @                0x0 (unknown)
  1. Pipeline driver crash

*** Aborted at 1718853165 (unix time) try "date -d @1718853165" if you are using GNU date ***
PC: @          0x4fcb438 starrocks::serde::(anonymous namespace)::read_raw()
*** SIGSEGV (@0x0) received by PID 49583 (TID 0x7f7b8404d700) from PID 0; stack trace: ***
    @          0x67c6d22 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f7d53d1005b os::Linux::chained_handler()
    @     0x7f7d53d1512b JVM_handle_linux_signal
    @     0x7f7d53d088c8 signalHandler()
    @     0x7f7d52ea35d0 (unknown)
    @          0x4fcb438 starrocks::serde::(anonymous namespace)::read_raw()
    @          0x4fd1187 starrocks::ColumnVisitorMutableAdapter<>::visit()
    @          0x2ba8b1c starrocks::ColumnFactory<>::accept_mutable()
    @          0x4fd08f8 starrocks::serde::ColumnArraySerde::deserialize()
    @          0x4fd0b6d starrocks::ColumnVisitorMutableAdapter<>::visit()
    @          0x345e00c starrocks::ColumnFactory<>::accept_mutable()
    @          0x4fd08f8 starrocks::serde::ColumnArraySerde::deserialize()
    @          0x4fd380a starrocks::serde::ProtobufChunkDeserializer::deserialize()
    @          0x2da5660 starrocks::DataStreamRecvr::SenderQueue::_deserialize_chunk()
    @          0x2daa08d starrocks::DataStreamRecvr::PipelineSenderQueue::get_chunk()
    @          0x2d7c593 starrocks::DataStreamRecvr::get_chunk_for_pipeline()
    @          0x37a44aa starrocks::pipeline::ExchangeSourceOperator::pull_chunk()
    @          0x3868146 starrocks::pipeline::PipelineDriver::process()
    @          0x385a70e starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x2e7d84c starrocks::ThreadPool::dispatch_thread()
    @          0x2e774ca starrocks::Thread::supervise_thread()
    @     0x7f7d52e9bdd5 start_thread
    @     0x7f7d5229dead __clone
    @                0x0 (unknown)
*** Aborted at 1721870966 (unix time) try "date -d @1721870966" if you are using GNU date ***
PC: @          0x2c3cfd8 starrocks::BinaryColumnBase<>::append_selective()
*** SIGSEGV (@0x100000006) received by PID 167194 (TID 0x7f8171b25700) from PID 6; stack trace: ***
    @          0x67c6d22 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f832aa0ce9b os::Linux::chained_handler()
    @     0x7f832aa11a5d JVM_handle_linux_signal
    @     0x7f832aa04858 signalHandler()
    @     0x7f8329eec5d0 (unknown)
    @          0x2c3cfd8 starrocks::BinaryColumnBase<>::append_selective()
    @          0x345d9dd starrocks::NullableColumn::append_selective()
    @          0x344219a starrocks::Chunk::append_selective()
    @          0x3aee96b starrocks::pipeline::ExchangeSinkOperator::Channel::add_rows_selective()
    @          0x3aefe75 starrocks::pipeline::ExchangeSinkOperator::push_chunk()
    @          0x3868b53 starrocks::pipeline::PipelineDriver::process()
    @          0x385a70e starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x2e7d84c starrocks::ThreadPool::dispatch_thread()
    @          0x2e774ca starrocks::Thread::supervise_thread()
    @     0x7f8329ee4dd5 start_thread
    @     0x7f83292e6ead __clone
    @                0x0 (unknown)
  1. Bad page: checksum mismatch

invalid encoding type:8

Or

num of element information corrupted, _num_element_after_padding:262721024, _num_elements:8

Or

Bad page: checksum mismatch (actual=2952600861 vs expect=1277960), file=/data/starRocks/storage/data/365/15004/2129610567/0200000000382bb9cd41910b6e0b0d512336bca4efb70dad_0.dat
  1. binary 类型导入 crash

    @         0x25715146 google::(anonymous namespace)::HandleSignal(int, siginfo_t*, void*)
    @     0x7f02beecaee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x25714989 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f02bee73520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @     0x7f02beec79fc pthread_kill
    @     0x7f02bee73476 raise
    @     0x7f02bee597f3 abort
    @     0x7f02bee5971b (/usr/lib/x86_64-linux-gnu/libc.so.6+0x2871a)
    @     0x7f02bee6ae96 __assert_fail
    @         0x1588d1d1 starrocks::JsonColumn* down_cast<starrocks::JsonColumn*, starrocks::Column>(starrocks::Column*)
    @         0x1f8ce798 starrocks::add_native_json_column(starrocks::Column*, starrocks::TypeDescriptor const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, simdjson::fallback::ondemand::value*)
    @         0x1f8a6807 starrocks::add_adaptive_nullable_native_json_column(starrocks::Column*, starrocks::TypeDescriptor const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, simdjson::fallback::ondemand::value*)
    @         0x1f8ab901 starrocks::add_adaptive_nullable_column(starrocks::Column*, starrocks::TypeDescriptor const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, simdjson::fallback::ondemand::value*)
    @         0x1f8aca75 starrocks::add_adaptive_nullable_column(starrocks::Column*, starrocks::TypeDescriptor const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, simdjson::fallback::ondemand::value*, bool)
    @         0x1f7b2a14 starrocks::JsonReader::_construct_column(simdjson::fallback::ondemand::value&, starrocks::Column*, starrocks::TypeDescriptor const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    @         0x1f7ae66c starrocks::JsonReader::_construct_row_with_jsonpath(simdjson::fallback::ondemand::object*, starrocks::Chunk*)
    @         0x1f7aef56 starrocks::JsonReader::_construct_row(simdjson::fallback::ondemand::object*, starrocks::Chunk*)
    @         0x1f7cd0ad starrocks::Status starrocks::JsonReader::_read_rows<starrocks::JsonArrayParser>(starrocks::Chunk*, int, int*)
    @         0x1f7ab7d8 starrocks::JsonReader::read_chunk(starrocks::Chunk*, int)
    @         0x1f7a4966 starrocks::JsonScanner::get_next()
    @         0x1f5badf9 starrocks::connector::FileDataSource::get_next(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*)
    @         0x1f99157c starrocks::ConnectorScanner::get_next(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*)
    @         0x1f98748c starrocks::ConnectorScanNode::_scanner_thread(starrocks::ConnectorScanner*)
    @         0x1f985744 starrocks::ConnectorScanNode::_submit_streaming_load_scanner(starrocks::ConnectorScanner*, bool)::{lambda()#1}::operator()() const
    @         0x1f98ad32 void std::__invoke_impl<void, starrocks::ConnectorScanNode::_submit_streaming_load_scanner(starrocks::ConnectorScanner*, bool)::{lambda()#1}&>(std::__invoke_other, starrocks::ConnectorScanNode::_submit_streaming_load_scanner(starrocks::ConnectorScanner*, b<DA>^A
    @         0x1f98a7dc std::enable_if<is_invocable_r_v<void, starrocks::ConnectorScanNode::_submit_streaming_load_scanner(starrocks::ConnectorScanner*, bool)::{lambda()#1}&>, void>::type std::__invoke_r<void, starrocks::ConnectorScanNode::_submit_streaming_load_scanner(starrocks<DA>^A
    @         0x1f98a3d5 std::_Function_handler<void (), starrocks::ConnectorScanNode::_submit_streaming_load_scanner(starrocks::ConnectorScanner*, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
    @         0x13abdcd2 std::function<void ()>::operator()() const
    @         0x20abf806 starrocks::FunctionRunnable::run()
    @         0x20abb726 starrocks::ThreadPool::dispatch_thread()
    @         0x20adadc8 void std::__invoke_impl<void, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(std::__invoke_memfun_deref, void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&)
    @         0x20ada869 std::__invoke_result<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>::type std::__invoke<void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&>(void (starrocks::ThreadPool::*&)(), starrocks::ThreadPool*&)
  1. 带有排序列的表,select * 结果行数不对

建表语句中有 order by

 ORDER BY (`shop_id`, `bday_date`)

没有谓词,但是 ShortKeyFilterRows 过滤掉了一些数据

                 - ShortKeyFilterRows: 72
                   - __MAX_OF_ShortKeyFilterRows: 72
                   - __MIN_OF_ShortKeyFilterRows: 0
  1. 磁盘间 Balance 导致已删除的文件无法清理

已删除的 Tablet 目录一直存在无法清理,导致磁盘空间膨胀。

  1. 磁盘间迁移频繁失败

同一个 tablet 磁盘间迁移频繁失败

  1. JAVA UDF 内存泄漏

*** SIGSEGV (@0x228) received by PID 1463834 (TID 0x7f75201bc640) from PID 552; stack trace: ***
    @     0x7f767c211ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @          0xc55e3a9 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f767cee101d os::Linux::chained_handler(int, siginfo_t*, void*)
    @     0x7f767cee5d49 JVM_handle_linux_signal
    @     0x7f767ced91f8 signalHandler(int, siginfo_t*, void*)
    @     0x7f767c1ba520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @     0x7f767ccb7924 jni_invoke_nonstatic(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .constprop.233]
    @     0x7f767ccc1579 jni_CallLongMethodV
    @          0x56d9f6f JNIEnv_::CallLongMethod(_jobject*, _jmethodID*, ...)
    @          0x8a30313 starrocks::append_jvalue(starrocks::MethodTypeDescriptor, starrocks::Column*, jvalue)
    @          0x78c5cbd starrocks::JavaUDTFFunction::process(starrocks::RuntimeState*, starrocks::TableFunctionState*) const
    @          0x5b2514e starrocks::pipeline::TableFunctionOperator::_process_table_function(starrocks::RuntimeState*)
    @          0x5b266ea starrocks::pipeline::TableFunctionOperator::pull_chunk(starrocks::RuntimeState*)
    @          0x5abff76 starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int)
    @          0x5aad0db starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x8b62fd4 starrocks::ThreadPool::dispatch_thread()
    @          0x8b5c0f9 starrocks::Thread::supervise_thread(void*)
    @     0x7f767c20cac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f767c29e850 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
  1. array_distinct crash

*** Aborted at 1726191146 (unix time) try "date -d @1726191146" if you are using GNU date ***
PC: @          0x99a52e4 starrocks::ArrayFunctions::array_distinct_any_type(starrocks::FunctionContext*, std::vector<std::shared_ptr<starrocks::Column>, std::allocator<std::shared_ptr<starrocks::Column> > > const&)
*** SIGSEGV (@0x7f972945cb28) received by PID 3508434 (TID 0x7f9729c5c640) from PID 692439848; stack trace: ***
    @     0x7f98781e3ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @          0xcf22069 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f987818c520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @          0x99a52e4 starrocks::ArrayFunctions::array_distinct_any_type(starrocks::FunctionContext*, std::vector<std::shared_ptr<starrocks::Column>, std::al
locator<std::shared_ptr<starrocks::Column> > > const&)
    @          0x81461e2 starrocks::VectorizedFunctionCallExpr::evaluate_checked(starrocks::ExprContext*, starrocks::Chunk*)
    @          0x5a40203 starrocks::ExprContext::evaluate(starrocks::Expr*, starrocks::Chunk*, unsigned char*)
    @          0x8145f85 starrocks::VectorizedFunctionCallExpr::evaluate_checked(starrocks::ExprContext*, starrocks::Chunk*)
    @          0x5a40203 starrocks::ExprContext::evaluate(starrocks::Expr*, starrocks::Chunk*, unsigned char*)
    @          0x5a40712 starrocks::ExprContext::evaluate(starrocks::Chunk*, unsigned char*)
    @          0x7d914fc starrocks::pipeline::ProjectOperator::push_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk> const&)
    @          0x5a8f0e3 starrocks::pipeline::PipelineDriver::process(starrocks::RuntimeState*, int)
    @          0x5b1e370 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
    @          0x9072432 starrocks::ThreadPool::dispatch_thread()
    @          0x906a6f9 starrocks::Thread::supervise_thread(void*)
    @     0x7f98781deac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f9878270850 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
  1. Rocksdb Tombstone 占用大量 CPU

--16.31%--rocksdb::MemTable::NewRangeTombstoneIterator
          |          
           --15.92%--rocksdb::FragmentedRangeTombstoneList::FragmentedRangeTombstoneList
                     |          
                      --13.88%--rocksdb::FragmentedRangeTombstoneList::FragmentTombstones
                                |          
                                |--6.38%--rocksdb::FragmentedRangeTombstoneList::FragmentTombstones(std::unique_ptr<rocksdb::InternalIteratorBase<rocksdb::Slice>, std::default_delete<rocksdb::InternalIteratorBase<rocksdb::Slice> > >, rocksdb::InternalKeyComparator const&, bool, std::vector<unsigned long, std::allocator<unsigned long> > const&)::{lambda(rocksdb::Slice const&)#1}::operator()
                                |          |          
                                |          |--2.28%--std::_Rb_tree<unsigned long, unsigned long, std::_Identity<unsigned long>, std::less<unsigned long>, std::allocator<unsigned long> >::_M_get_insert_hint_unique_pos
                                |          |  
                                |           --0.68%--__memcmp_sse4_1
                                |          
                                |--2.36%--operator new
                                |          |          
                                |           --1.67%--my_malloc
                                |                     |          
                                |                      --0.51%--je_malloc_default
                                |          
                                |--1.77%--std::_Rb_tree<rocksdb::ParsedInternalKey, rocksdb::ParsedInternalKey, std::_Identity<rocksdb::ParsedInternalKey>, rocksdb::ParsedInternalKeyComparator, std::allocator<rocksdb::ParsedInternalKey> >::_M_emplace_unique<rocksdb::Slice&, unsigned long&, rocksdb::ValueType>
                                |          |          
                                |           --1.34%--rocksdb::InternalKeyComparator::Compare
                                |                     |          
                                |                      --0.55%--__memcmp_sse4_1
                                |          
                                 --0.70%--std::_Rb_tree_insert_and_rebalance