版本:3.3.16 RELEASE (build a81c69f)
环境:K8s 存算分离模式
复现条件:高并发 Load 场景
崩溃堆栈:
*** SIGSEGV (@0x10) received by PID 27 ***
std::_Sp_counted_base<>::_M_release()
opentelemetry::v1::trace::Scope::Scope(shared_ptr const&)
starrocks::LoadChannel::open(…)
starrocks::LoadChannelMgr::open(…)
brpc::policy::ProcessRpcRequest(…)
附带影响:崩溃后 Data Cache 中留下损坏的数据页,
后续查询报 “invalid size_of_elem:0”
详细日志信息:
[root@k8s-master01 tmp]# kubectl logs starrocks-cn-0 -n enginemanager-vncn --previous --tail 800
[Sun Mar 29 11:57:50 CST 2026] Process conf file cn.conf …
[Sun Mar 29 11:57:50 CST 2026] Add myself (starrocks-cn-0.starrocks-cn-search.enginemanager-vncn.svc.cluster.local:9050) into FE …
ERROR 1064 (HY000) at line 1: Compute node already exists with same host starrocks-cn-0.starrocks-cn-search.enginemanager-vncn.svc.cluster.local and port 9050
[Sun Mar 29 11:57:50 CST 2026] run start_cn.sh
/opt/starrocks/cn_entrypoint.sh: line 187: 27 Segmentation fault (core dumped) $STARROCKS_HOME/bin/start_cn.sh $addition_args
[Mon Mar 30 16:15:19 CST 2026] Last 50 lines of cn.INFO …
I20260330 16:15:15.596298 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94f6b107-2c10-11f1-b157-be72ea064993, txn_id: 1098568, add chunk time(ms)/wait lock time(ms)/num: {2009266:(0)(0)(1)} {14004:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.601381 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94f6b107-2c10-11f1-b157-be72ea064993, txn_id: 1098568, add chunk time(ms)/wait lock time(ms)/num: {2009266:(0)(0)(1)} {14004:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.606460 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94f6b107-2c10-11f1-b157-be72ea064993, txn_id: 1098568, add chunk time(ms)/wait lock time(ms)/num: {2009266:(0)(0)(1)} {14004:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.613127 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94fccbec-2c10-11f1-8740-562a3c769ede, txn_id: 1098569, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.618228 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94fccbec-2c10-11f1-8740-562a3c769ede, txn_id: 1098569, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.618314 139993506649664 tablet_sink_index_channel.cpp:861] OlapTableSink txn_id: 1098569 load_id: 94fccbec-2c10-11f1-8740-562a3c769ede commit 22 tablets: 1318321,1318324,1318326,1318327,1318330,1318334,1318337,1318339,1318344,1318349,1318351,1318353,1318355,1318357,1318359,1318360,1318365,1318368,1318370,1318372,1318373,1318374
I20260330 16:15:15.618330 139993506649664 tablet_sink_index_channel.cpp:861] OlapTableSink txn_id: 1098569 load_id: 94fccbec-2c10-11f1-8740-562a3c769ede commit 21 tablets: 1318316,1318317,1318320,1318325,1318328,1318333,1318335,1318341,1318343,1318345,1318346,1318348,1318358,1318361,1318362,1318364,1318366,1318369,1318375,1318377,1318378
I20260330 16:15:15.618343 139993506649664 tablet_sink_index_channel.cpp:861] OlapTableSink txn_id: 1098569 load_id: 94fccbec-2c10-11f1-8740-562a3c769ede commit 21 tablets: 1318315,1318318,1318319,1318322,1318323,1318329,1318331,1318332,1318336,1318338,1318340,1318342,1318347,1318350,1318352,1318354,1318356,1318363,1318367,1318371,1318376
I20260330 16:15:15.623462 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94fccbec-2c10-11f1-8740-562a3c769ede, txn_id: 1098569, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(8)(0)(1)}
I20260330 16:15:15.628534 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94fccbec-2c10-11f1-8740-562a3c769ede, txn_id: 1098569, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(7)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.633605 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94fccbec-2c10-11f1-8740-562a3c769ede, txn_id: 1098569, add chunk time(ms)/wait lock time(ms)/num: {14004:(7)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.650608 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 950382aa-2c10-11f1-a47c-f64cd03fcf59, txn_id: 1098571, add chunk time(ms)/wait lock time(ms)/num: {1598005:(0)(0)(1)} {2009266:(0)(0)(1)} {14004:(0)(0)(1)}
I20260330 16:15:15.650627 139993506649664 tablet_sink_index_channel.cpp:861] OlapTableSink txn_id: 1098571 load_id: 950382aa-2c10-11f1-a47c-f64cd03fcf59 commit 3 tablets: 11681,11686,11688
I20260330 16:15:15.655727 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 950382aa-2c10-11f1-a47c-f64cd03fcf59, txn_id: 1098571, add chunk time(ms)/wait lock time(ms)/num: {1598005:(0)(0)(1)} {2009266:(0)(0)(1)} {14004:(0)(0)(1)}
I20260330 16:15:15.655737 139993506649664 tablet_sink_index_channel.cpp:861] OlapTableSink txn_id: 1098571 load_id: 950382aa-2c10-11f1-a47c-f64cd03fcf59 commit 4 tablets: 11682,11685,11687,11689
I20260330 16:15:15.660877 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 950382aa-2c10-11f1-a47c-f64cd03fcf59, txn_id: 1098571, add chunk time(ms)/wait lock time(ms)/num: {1598005:(0)(0)(1)} {2009266:(4)(0)(1)} {14004:(7)(0)(1)}
I20260330 16:15:15.681158 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94e1a2a8-2c10-11f1-a47c-f64cd03fcf59, txn_id: 1098565, add chunk time(ms)/wait lock time(ms)/num: {2009266:(5)(0)(8)} {1598005:(9)(0)(8)} {14004:(11)(0)(9)}
I20260330 16:15:15.686257 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94e1a2a8-2c10-11f1-a47c-f64cd03fcf59, txn_id: 1098565, add chunk time(ms)/wait lock time(ms)/num: {2009266:(5)(0)(8)} {1598005:(13)(0)(8)} {14004:(7)(0)(8)}
I20260330 16:15:15.691617 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94e1a2a8-2c10-11f1-a47c-f64cd03fcf59, txn_id: 1098565, add chunk time(ms)/wait lock time(ms)/num: {2009266:(5)(0)(8)} {1598005:(12)(0)(9)} {14004:(6)(0)(8)}
I20260330 16:15:15.697043 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 950a125d-2c10-11f1-8740-562a3c769ede, txn_id: 1098572, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.702126 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 950a125d-2c10-11f1-8740-562a3c769ede, txn_id: 1098572, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.707196 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 950a125d-2c10-11f1-8740-562a3c769ede, txn_id: 1098572, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.712381 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 950a125d-2c10-11f1-8740-562a3c769ede, txn_id: 1098572, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.720611 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94e1a2a8-2c10-11f1-a47c-f64cd03fcf59, txn_id: 1098565, add chunk time(ms)/wait lock time(ms)/num: {2009266:(13)(0)(9)} {1598005:(7)(0)(9)} {14004:(16)(0)(9)}
I20260330 16:15:15.720663 139993506649664 tablet_sink_index_channel.cpp:861] OlapTableSink txn_id: 1098572 load_id: 950a125d-2c10-11f1-8740-562a3c769ede commit 22 tablets: 1318321,1318324,1318326,1318327,1318330,1318334,1318337,1318339,1318344,1318349,1318351,1318353,1318355,1318357,1318359,1318360,1318365,1318368,1318370,1318372,1318373,1318374
I20260330 16:15:15.720677 139993506649664 tablet_sink_index_channel.cpp:861] OlapTableSink txn_id: 1098572 load_id: 950a125d-2c10-11f1-8740-562a3c769ede commit 21 tablets: 1318316,1318317,1318320,1318325,1318328,1318333,1318335,1318341,1318343,1318345,1318346,1318348,1318358,1318361,1318362,1318364,1318366,1318369,1318375,1318377,1318378
I20260330 16:15:15.720687 139993506649664 tablet_sink_index_channel.cpp:861] OlapTableSink txn_id: 1098572 load_id: 950a125d-2c10-11f1-8740-562a3c769ede commit 21 tablets: 1318315,1318318,1318319,1318322,1318323,1318329,1318331,1318332,1318336,1318338,1318340,1318342,1318347,1318350,1318352,1318354,1318356,1318363,1318367,1318371,1318376
I20260330 16:15:15.725747 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 94e1a2a8-2c10-11f1-a47c-f64cd03fcf59, txn_id: 1098565, add chunk time(ms)/wait lock time(ms)/num: {2009266:(14)(0)(9)} {1598005:(8)(0)(8)} {14004:(3)(0)(8)}
I20260330 16:15:15.730847 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 950a125d-2c10-11f1-8740-562a3c769ede, txn_id: 1098572, add chunk time(ms)/wait lock time(ms)/num: {14004:(8)(0)(1)} {2009266:(7)(0)(1)} {1598005:(8)(0)(1)}
I20260330 16:15:15.781730 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9517ce00-2c10-11f1-8740-562a3c769ede, txn_id: 1098574, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.786835 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9517ce00-2c10-11f1-8740-562a3c769ede, txn_id: 1098574, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.791904 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9517ce00-2c10-11f1-8740-562a3c769ede, txn_id: 1098574, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.797005 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9517ce00-2c10-11f1-8740-562a3c769ede, txn_id: 1098574, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.802074 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9517ce00-2c10-11f1-8740-562a3c769ede, txn_id: 1098574, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.836479 139993750038080 internal_service.cpp:525] cancel fragment, fragment_instance_id=9520f5c4-2c10-11f1-8740-562a3c769ee2, reason: QueryFinished
I20260330 16:15:15.852642 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9522a376-2c10-11f1-8740-562a3c769ede, txn_id: 1098575, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.857719 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9522a376-2c10-11f1-8740-562a3c769ede, txn_id: 1098575, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.862788 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9522a376-2c10-11f1-8740-562a3c769ede, txn_id: 1098575, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.867857 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9522a376-2c10-11f1-8740-562a3c769ede, txn_id: 1098575, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.872927 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9522a376-2c10-11f1-8740-562a3c769ede, txn_id: 1098575, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.932928 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 952eff88-2c10-11f1-8740-562a3c769ede, txn_id: 1098576, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.938030 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 952eff88-2c10-11f1-8740-562a3c769ede, txn_id: 1098576, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.943093 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 952eff88-2c10-11f1-8740-562a3c769ede, txn_id: 1098576, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.948156 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 952eff88-2c10-11f1-8740-562a3c769ede, txn_id: 1098576, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.953218 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 952eff88-2c10-11f1-8740-562a3c769ede, txn_id: 1098576, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:15.959619 139993758430784 internal_service.cpp:525] cancel fragment, fragment_instance_id=95308636-2c10-11f1-a47c-f64cd03fcf5c, reason: QueryFinished
I20260330 16:15:16.003067 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9539d4f9-2c10-11f1-8740-562a3c769ede, txn_id: 1098577, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:16.008163 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9539d4f9-2c10-11f1-8740-562a3c769ede, txn_id: 1098577, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
I20260330 16:15:16.013236 139993506649664 tablet_sink_sender.cpp:353] Olap table sink statistics. load_id: 9539d4f9-2c10-11f1-8740-562a3c769ede, txn_id: 1098577, add chunk time(ms)/wait lock time(ms)/num: {14004:(0)(0)(1)} {2009266:(0)(0)(1)} {1598005:(0)(0)(1)}
[Mon Mar 30 16:15:19 CST 2026] Last 50 lines of cn.out …
I20260330 16:15:16.018301 139993506649664 tablet_sink_sender.cpp:353] Olapstart time: Sun Mar 29 11:57:50 CST 2026, server uptime: 11:57:50 up 47 days, 23:48, 0 users, load average: 5.88, 6.38, 6.54
3.3.16 RELEASE (build a81c69f)
query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000
tracker:process consumption: 16561330952
tracker:jemalloc_metadata consumption: 1243583712
tracker:query_pool consumption: 31161552
tracker:query_pool/connector_scan consumption: 0
tracker:load consumption: 36944
tracker:metadata consumption: 1463860754
tracker:tablet_metadata consumption: 725102
tracker:rowset_metadata consumption: 0
tracker:segment_metadata consumption: 194110641
tracker:column_metadata consumption: 1269025011
tracker:tablet_schema consumption: 725102
tracker:segment_zonemap consumption: 172677121
tracker:short_key_index consumption: 1633447
tracker:column_zonemap_index consumption: 329010683
tracker:ordinal_index consumption: 242964552
tracker:bitmap_index consumption: 0
tracker:bloom_filter_index consumption: 0
tracker:compaction consumption: 0
tracker:schema_change consumption: 0
tracker:column_pool consumption: 0
tracker:page_cache consumption: 11205316544
tracker:jit_cache consumption: 11912
tracker:update consumption: 20816849
tracker:chunk_allocator consumption: 0
tracker:passthrough consumption: 0
tracker:clone consumption: 0
tracker:consistency consumption: 0
tracker:datacache consumption: 540103848
tracker:replication consumption: 0
*** Aborted at 1774858515 (unix time) try “date -d @1774858515” if you are using GNU date ***
PC: @ 0x52efcc7 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
*** SIGSEGV (@0x10) received by PID 27 (TID 0x7f51f6579640) from PID 16; stack trace: ***
@ 0x7f5374d40ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
@ 0xa2ba469 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
@ 0x7f5374ce9520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
@ 0x52efcc7 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
@ 0x6b4d9b1 opentelemetry::v1::trace::Scope::Scope(opentelemetry::v1::nostd::shared_ptropentelemetry::v1::trace::Span const&)
@ 0x8961eb0 starrocks::LoadChannel::open(brpc::Controller*, starrocks::PTabletWriterOpenRequest const&, starrocks::PTabletWriterOpenResult*, google::protobuf::Closure*)
@ 0x895b6c0 starrocks::LoadChannelMgr::open(brpc::Controller*, starrocks::PTabletWriterOpenRequest const&, starrocks::PTabletWriterOpenResult*, google::protobuf::Closure*)
@ 0xa59bb83 brpc::policy::ProcessRpcRequest(brpc::InputMessageBase*)
@ 0xa4c175b brpc::ProcessInputMessage(void*)
@ 0xa4c2ba4 brpc::InputMessenger::OnNewMessages(brpc::Socket*)
@ 0xa4ff4c2 brpc::Socket::ProcessEvent(void*)
@ 0xa476657 bthread::TaskGroup::task_runner(long)
@ 0xa45f641 bthread_make_fcontext
[1774858515.525][thread:139990002013760] je_mallctl execute purge success
[1774858515.525][thread:139990002013760] je_mallctl execute dontdump success