be节点挂掉

为了更快的定位您的问题,请提供以下信息,谢谢
【详述】be节点挂掉
【背景】做过哪些操作?
【业务影响】
【StarRocks版本】例如:3.0.2
【集群规模】例如:3fe(3 follower)+12be(2台follower混布)
【机器信息】8c32g
【联系方式】18840044216
【附件】
be.info
W0921 10:43:09.915359 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7256462], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.915369 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813291], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.915376 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813291], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
I0921 10:43:09.915527 31320 tablet_sink.cpp:1831] Olap table sink statistics. load_id: 826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, add chunk time(ms)/wait lock time(ms)/num: {7813291:(7)(0)(4)} {7256462:(10)(0)(5)} {63403348:(190)(0)(6)} {63403042:(17)(0)(4)} {10008:(10)(0)(5)} {7813337:(19)(0)(5)} {63405783:(40)(0)(6)} {10003:(42)(0)(5)} {63403080:(220)(0)(5)} {7813312:(5)(0)(4)} {42565:(123)(0)(6)} {7795613:(770)(0)(4)}
I0921 10:43:09.915536 31320 profile_report_worker.cpp:59] unregister_pipeline_load query_id=826e5915-5828-11ee-905b-00163e13470f, fragment_instance_id=826e5915-5828-11ee-905b-00163e13471f
I0921 10:43:09.915535 31326 tablet_sink.cpp:1831] Olap table sink statistics. load_id: 826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, add chunk time(ms)/wait lock time(ms)/num: {7813291:(193)(0)(4)} {7256462:(24)(0)(4)} {63403348:(245)(0)(5)} {63403042:(3)(0)(3)} {10008:(24)(0)(4)} {7813337:(30)(0)(4)} {63405783:(27)(0)(5)} {10003:(47)(0)(4)} {63403080:(120)(0)(4)} {7813312:(2)(0)(3)} {42565:(57)(0)(5)} {7795613:(630)(0)(3)}
I0921 10:43:09.915553 31326 profile_report_worker.cpp:59] unregister_pipeline_load query_id=826e5915-5828-11ee-905b-00163e13470f, fragment_instance_id=826e5915-5828-11ee-905b-00163e13471f
I0921 10:43:09.915673 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e1347aa -> 6a95c388-5828-11ee-905b-00163e1347aa, _num_uncancelled_sinkers 0, _is_finishing true, _num_remaining_eos 5
W0921 10:43:09.915782 31310 pipeline_driver.cpp:597] fragment_id 6a95c388-5828-11ee-905b-00163e13483b driver query_id=6a95c388-5828-11ee-905b-00163e13470f fragment_id=6a95c388-5828-11ee-905b-00163e13483b driver=0x7f97dac94710, status=INPUT_EMPTY, operator-chain: [exchange_source_99_0x7f9a159b9110(O) -> local_sort_sink_100_0x7f97dac93090(X)] cancels operator local_sort_sink_100_0x7f97dac93090(X) with finished error runtime state is cancelled
W0921 10:43:09.915915 31310 pipeline_driver.cpp:597] fragment_id 6a95c388-5828-11ee-905b-00163e13483b driver query_id=6a95c388-5828-11ee-905b-00163e13470f fragment_id=6a95c388-5828-11ee-905b-00163e13483b driver=0x7f97dacad110, status=INPUT_EMPTY, operator-chain: [exchange_source_99_0x7f97dac95890(O) -> local_sort_sink_100_0x7f97dacab810(X)] cancels operator local_sort_sink_100_0x7f97dacab810(X) with finished error runtime state is cancelled
W0921 10:43:09.916024 31310 pipeline_driver.cpp:597] fragment_id 6a95c388-5828-11ee-905b-00163e13483b driver query_id=6a95c388-5828-11ee-905b-00163e13470f fragment_id=6a95c388-5828-11ee-905b-00163e13483b driver=0x7f97dadd4e10, status=INPUT_EMPTY, operator-chain: [exchange_source_99_0x7f97dacad610(O) -> local_sort_sink_100_0x7f97dadd3790(X)] cancels operator local_sort_sink_100_0x7f97dadd3790(X) with finished error runtime state is cancelled
W0921 10:43:09.916122 31310 pipeline_driver.cpp:597] fragment_id 6a95c388-5828-11ee-905b-00163e13483b driver query_id=6a95c388-5828-11ee-905b-00163e13470f fragment_id=6a95c388-5828-11ee-905b-00163e13483b driver=0x7f97db040510, status=INPUT_EMPTY, operator-chain: [exchange_source_99_0x7f97dadd5310(O) -> local_sort_sink_100_0x7f97dafd7e90(X)] cancels operator local_sort_sink_100_0x7f97dafd7e90(X) with finished error runtime state is cancelled
W0921 10:43:09.916221 31310 pipeline_driver.cpp:597] fragment_id 6a95c388-5828-11ee-905b-00163e13483b driver query_id=6a95c388-5828-11ee-905b-00163e13470f fragment_id=6a95c388-5828-11ee-905b-00163e13483b driver=0x7f97db0eb010, status=INPUT_EMPTY, operator-chain: [exchange_source_99_0x7f97db040c90(X) -> local_sort_sink_100_0x7f97db044610(X)] cancels operator local_sort_sink_100_0x7f97db044610(X) with finished error runtime state is cancelled
I0921 10:43:09.916253 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e13483b -> 6a95c388-5828-11ee-905b-00163e13483b, _num_uncancelled_sinkers 4, _is_finishing false, _num_remaining_eos 60
I0921 10:43:09.916265 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e13483b -> 6a95c388-5828-11ee-905b-00163e13483b, _num_uncancelled_sinkers 3, _is_finishing false, _num_remaining_eos 60
I0921 10:43:09.916273 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e13483b -> 6a95c388-5828-11ee-905b-00163e13483b, _num_uncancelled_sinkers 2, _is_finishing false, _num_remaining_eos 60
I0921 10:43:09.916281 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e13483b -> 6a95c388-5828-11ee-905b-00163e13483b, _num_uncancelled_sinkers 1, _is_finishing false, _num_remaining_eos 60
I0921 10:43:09.916289 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e13483b -> 6a95c388-5828-11ee-905b-00163e13483b, _num_uncancelled_sinkers 0, _is_finishing true, _num_remaining_eos 60
I0921 10:43:09.916296 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e134794 -> 6a95c388-5828-11ee-905b-00163e134794, _num_uncancelled_sinkers 4, _is_finishing false, _num_remaining_eos 3
I0921 10:43:09.916302 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e134794 -> 6a95c388-5828-11ee-905b-00163e134794, _num_uncancelled_sinkers 3, _is_finishing false, _num_remaining_eos 3
I0921 10:43:09.916318 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e134794 -> 6a95c388-5828-11ee-905b-00163e134794, _num_uncancelled_sinkers 2, _is_finishing false, _num_remaining_eos 3
I0921 10:43:09.916324 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e134794 -> 6a95c388-5828-11ee-905b-00163e134794, _num_uncancelled_sinkers 1, _is_finishing false, _num_remaining_eos 3
I0921 10:43:09.916329 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e134794 -> 6a95c388-5828-11ee-905b-00163e134794, _num_uncancelled_sinkers 0, _is_finishing true, _num_remaining_eos 3
W0921 10:43:09.920604 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7795613], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920619 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7795613], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920631 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[42565], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920642 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[42565], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920648 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813312], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920655 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813312], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920667 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403080], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920675 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403080], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920681 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[10003], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920688 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[10003], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920693 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63405783], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920701 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63405783], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920706 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813337], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920712 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813337], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920729 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[10008], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920737 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[10008], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920742 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403042], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920749 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403042], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920755 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403348], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920761 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403348], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920768 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7256462], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920774 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7256462], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920783 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813291], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.920790 31326 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813291], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
I0921 10:43:09.920950 31320 tablet_sink.cpp:1831] Olap table sink statistics. load_id: 826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, add chunk time(ms)/wait lock time(ms)/num: {7813291:(7)(0)(4)} {7256462:(10)(0)(5)} {63403348:(190)(0)(6)} {63403042:(17)(0)(4)} {10008:(10)(0)(5)} {7813337:(19)(0)(5)} {63405783:(40)(0)(6)} {10003:(42)(0)(5)} {63403080:(220)(0)(5)} {7813312:(5)(0)(4)} {42565:(123)(0)(6)} {7795613:(770)(0)(4)}
I0921 10:43:09.920959 31320 profile_report_worker.cpp:59] unregister_pipeline_load query_id=826e5915-5828-11ee-905b-00163e13470f, fragment_instance_id=826e5915-5828-11ee-905b-00163e13471f
I0921 10:43:09.920959 31326 tablet_sink.cpp:1831] Olap table sink statistics. load_id: 826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, add chunk time(ms)/wait lock time(ms)/num: {7813291:(193)(0)(4)} {7256462:(24)(0)(4)} {63403348:(245)(0)(5)} {63403042:(3)(0)(3)} {10008:(24)(0)(4)} {7813337:(30)(0)(4)} {63405783:(27)(0)(5)} {10003:(47)(0)(4)} {63403080:(120)(0)(4)} {7813312:(2)(0)(3)} {42565:(57)(0)(5)} {7795613:(630)(0)(3)}
I0921 10:43:09.920971 31326 profile_report_worker.cpp:59] unregister_pipeline_load query_id=826e5915-5828-11ee-905b-00163e13470f, fragment_instance_id=826e5915-5828-11ee-905b-00163e13471f
W0921 10:43:09.924400 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[7795613], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924443 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[42565], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924449 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[7813312], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924455 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[63403080], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924459 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[10003], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924464 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[63405783], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924469 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[7813337], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924474 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[10008], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924477 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[63403042], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924481 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[63403348], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924486 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[7256462], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.924502 31310 tablet_sink.cpp:1634] close channel failed. channel_name=NodeChannel[7813291], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
I0921 10:43:09.924600 31310 profile_report_worker.cpp:59] unregister_pipeline_load query_id=826e5915-5828-11ee-905b-00163e13470f, fragment_instance_id=826e5915-5828-11ee-905b-00163e13471f
I0921 10:43:09.924613 31310 sink_buffer.cpp:204] fragment_instance_id 6a95c388-5828-11ee-905b-00163e134833 -> 6a95c388-5828-11ee-905b-00163e134833, _num_uncancelled_sinkers 0, _is_finishing true, _num_remaining_eos 4
I0921 10:43:09.924800 31313 profile_report_worker.cpp:59] unregister_pipeline_load query_id=6a95c388-5828-11ee-905b-00163e13470f, fragment_instance_id=6a95c388-5828-11ee-905b-00163e134833
I0921 10:43:09.924933 31324 profile_report_worker.cpp:59] unregister_pipeline_load query_id=6a95c388-5828-11ee-905b-00163e13470f, fragment_instance_id=6a95c388-5828-11ee-905b-00163e13483b
I0921 10:43:09.925623 31316 profile_report_worker.cpp:59] unregister_pipeline_load query_id=6a95c388-5828-11ee-905b-00163e13470f, fragment_instance_id=6a95c388-5828-11ee-905b-00163e1347aa
I0921 10:43:09.929343 31291 thrift_client.cpp:81] Error closing connection to: 172.16.4.127:9020, ignoring (write() send(): Broken pipe)
I0921 10:43:09.929666 26076 thrift_client.cpp:81] Error closing connection to: 172.16.4.127:9020, ignoring (write() send(): Broken pipe)
W0921 10:43:09.929765 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7795613], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929793 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[42565], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929816 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813312], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929823 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403080], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929831 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[10003], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929842 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63405783], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929849 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813337], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929857 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[10008], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929863 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403042], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929870 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403348], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929877 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7256462], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.929884 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813291], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
I0921 10:43:09.930066 31320 tablet_sink.cpp:1831] Olap table sink statistics. load_id: 826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, add chunk time(ms)/wait lock time(ms)/num: {7813291:(50)(0)(4)} {7256462:(8)(0)(4)} {63403348:(214)(0)(5)} {63403042:(5)(0)(4)} {10008:(25)(0)(4)} {7813337:(52)(0)(4)} {63405783:(30)(0)(5)} {10003:(15)(0)(4)} {63403080:(235)(0)(4)} {7813312:(2)(0)(3)} {42565:(208)(0)(5)} {7795613:(621)(0)(3)}
I0921 10:43:09.930078 31320 profile_report_worker.cpp:59] unregister_pipeline_load query_id=826e5915-5828-11ee-905b-00163e13470f, fragment_instance_id=826e5915-5828-11ee-905b-00163e13471f
I0921 10:43:09.930938 31291 pipeline_driver_executor.cpp:263] [Driver] Fail to report exec state due to query not found: fragment_instance_id=6a95c388-5828-11ee-905b-00163e13483b
I0921 10:43:09.931396 26076 pipeline_driver_executor.cpp:263] [Driver] Fail to report exec state due to query not found: fragment_instance_id=6a95c388-5828-11ee-905b-00163e134833
W0921 10:43:09.935153 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7795613], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935195 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[42565], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935204 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813312], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935210 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403080], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935217 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[10003], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935225 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63405783], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935235 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813337], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935242 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[10008], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935249 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403042], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935256 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[63403348], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935263 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7256462], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
W0921 10:43:09.935271 31320 tablet_sink.cpp:1797] close channel failed. channel_name=NodeChannel[7813291], load_info=load_id=826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, parallel=1, compress_type=2, error_msg=Cancelled by pipeline engine
I0921 10:43:09.935449 31320 tablet_sink.cpp:1831] Olap table sink statistics. load_id: 826e5915-5828-11ee-905b-00163e13470f, txn_id: 56997565, add chunk time(ms)/wait lock time(ms)/num: {7813291:(50)(0)(4)} {7256462:(8)(0)(4)} {63403348:(214)(0)(5)} {63403042:(5)(0)(4)} {10008:(25)(0)(4)} {7813337:(52)(0)(4)} {63405783:(30)(0)(5)} {10003:(15)(0)(4)} {63403080:(235)(0)(4)} {7813312:(2)(0)(3)} {42565:(208)(0)(5)} {7795613:(621)(0)(3)}
I0921 10:43:09.935459 31320 profile_report_worker.cpp:59] unregister_pipeline_load query_id=826e5915-5828-11ee-905b-00163e13470f, fragment_instance_id=826e5915-5828-11ee-905b-00163e13471f
I0921 10:43:09.936023 31320 profile_report_worker.cpp:59] unregister_pipeline_load query_id=826e5915-5828-11ee-905b-00163e13470f, fragment_instance_id=826e5915-5828-11ee-905b-00163e13471f
I0921 10:43:09.938985 26076 thrift_client.cpp:81] Error closing connection to: 172.16.4.127:9020, ignoring (write() send(): Broken pipe)
I0921 10:43:09.940610 26076 pipeline_driver_executor.cpp:263] [Driver] Fail to report exec state due to query not found: fragment_instance_id=826e5915-5828-11ee-905b-00163e13471f
I0921 10:43:09.968572 26076 thrift_client.cpp:81] Error closing connection to: 172.16.4.127:9020, ignoring (write() send(): Broken pipe)
I0921 10:43:09.978874 26076 pipeline_driver_executor.cpp:263] [Driver] Fail to report exec state due to query not found: fragment_instance_id=6a95c388-5828-11ee-905b-00163e1347aa
I0921 10:43:10.000552 31782 data_consumer.cpp:73] init kafka consumer with group id: loadobc_bi_refundInfoPO_dbaed023-3ccd-4fd9-8f81-b9376e3bedda
I0921 10:43:10.121501 31317 profile_report_worker.cpp:59] unregister_pipeline_load query_id=6a95c388-5828-11ee-905b-00163e13470f, fragment_instance_id=6a95c388-5828-11ee-905b-00163e134794
I0921 10:43:10.129325 26076 thrift_client.cpp:81] Error closing connection to: 172.16.4.127:9020, ignoring (write() send(): Broken pipe)
I0921 10:43:10.132009 26076 pipeline_driver_executor.cpp:263] [Driver] Fail to report exec state due to query not found: fragment_instance_id=6a95c388-5828-11ee-905b-00163e134794
W0921 10:43:09.925055 31325 mem_hook.cpp:266] large memory alloc: 2481305800 bytes, stack:
@ 0x4fe943b malloc
@ 0x8a56d85 operator new()
@ 0x30d9b3e std::vector<>::_M_range_insert<>()
@ 0x30dd0f4 starrocks::BinaryColumnBase<>::append()
@ 0x569a7ea starrocks::NullableColumn::append()
@ 0x567e7f2 starrocks::Chunk::append()
@ 0x32acf5f starrocks::ChunksSorterFullSort::_partial_sort()
@ 0x32ad441 starrocks::ChunksSorterFullSort::update()
@ 0x3410026 starrocks::pipeline::PartitionSortSinkOperator::push_chunk()
@ 0x310fef9 starrocks::pipeline::PipelineDriver::process()
@ 0x578d84b starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x50e5222 starrocks::ThreadPool::dispatch_thread()
@ 0x50dfd1a starrocks::thread::supervise_thread()
@ 0x7f9a36628ea5 start_thread
@ 0x7f9a35c439fd __clone
@ (nil) (unknown)
I0921 10:47:20.791985 28251 daemon.cpp:291] version 3.0.2 RELEASE (build c833698)

be挂掉贴下be.out文件

be crash可以按照这个排查下, [问题排查]BE Crash

*** Aborted at 1689060484 (unix time) try “date -d @1689060484” if you are using GNU date ***
PC: @ 0x485e8ac starrocks::SegmentIterator::_init_column_iterators<>()
*** SIGSEGV (@0x0) received by PID 26969 (TID 0x7f12efff3700) from PID 0; stack trace: ***
@ 0x62d7062 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f14b3661f2b os::Linux::chained_handler()
@ 0x7f14b3666b0d JVM_handle_linux_signal
@ 0x7f14b36598e8 signalHandler()
@ 0x7f14b2b41630 (unknown)
@ 0x485e8ac starrocks::SegmentIterator::_init_column_iterators<>()
@ 0x4851612 starrocks::SegmentIterator::_init()
@ 0x4851ac0 starrocks::SegmentIterator::do_get_next()
@ 0x48d3e22 starrocks::MaskMergeIterator::do_get_next()
@ 0x49355ec starrocks::RowsetMergerImpl<>::_do_merge_vertically()
@ 0x4936f24 starrocks::RowsetMergerImpl<>::do_merge()
@ 0x492a2cf starrocks::compaction_merge_rowsets()
@ 0x47da196 starrocks::TabletUpdates::_do_compaction()
@ 0x47db59a starrocks::TabletUpdates::compaction()
@ 0x4736053 starrocks::StorageEngine::_perform_update_compaction()
@ 0x49eb09e starrocks::StorageEngine::_update_compaction_thread_callback()
@ 0x8ad1ac0 execute_native_thread_routine
@ 0x7f14b2b39ea5 start_thread
@ 0x7f14b21549fd __clone
@ 0x0 (unknown)
start time: Wed Jul 12 11:56:35 CST 2023
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/starrocks/be/lib/jni-packages/starrocks-jdbc-bridge-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/starrocks/be/lib/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
3.0.2 RELEASE (build c833698)
query_id:995f9c8a-5828-11ee-b1a9-00163e0c0e88, fragment_instance:995f9c8a-5828-11ee-b1a9-00163e0c0e89
tracker:process consumption: 12888478448
tracker:query_pool consumption: 3448406864
tracker:load consumption: 35420432
tracker:metadata consumption: 1261557945
tracker:tablet_metadata consumption: 517130027
tracker:rowset_metadata consumption: 188933867
tracker:segment_metadata consumption: 175566936
tracker:column_metadata consumption: 379927115
tracker:tablet_schema consumption: 32774843
tracker:segment_zonemap consumption: 144115728
tracker:short_key_index consumption: 23590295
tracker:column_zonemap_index consumption: 70570491
tracker:ordinal_index consumption: 130699296
tracker:bitmap_index consumption: 18018992
tracker:bloom_filter_index consumption: 0
tracker:compaction consumption: 2500512
tracker:schema_change consumption: 0
tracker:column_pool consumption: 0
tracker:page_cache consumption: 0
tracker:update consumption: 3446366424
tracker:chunk_allocator consumption: 2150329976
tracker:clone consumption: 0
tracker:consistency consumption: 0
*** Aborted at 1695264190 (unix time) try “date -d @1695264190” if you are using GNU date ***
PC: @ 0x7f9a35c99ba5 __memcpy_ssse3_back
*** SIGSEGV (@0x0) received by PID 30963 (TID 0x7f9920737700) from PID 0; stack trace: ***
@ 0x62d7062 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7f9a37150f2b os::Linux::chained_handler()
@ 0x7f9a371559bc JVM_handle_linux_signal
@ 0x7f9a371488e8 signalHandler()
@ 0x7f9a36630630 (unknown)
@ 0x7f9a35c99ba5 __memcpy_ssse3_back
@ 0x30d96e2 std::vector<>::_M_range_insert<>()
@ 0x30dd0f4 starrocks::BinaryColumnBase<>::append()
@ 0x569a7ea starrocks::NullableColumn::append()
@ 0x567e7f2 starrocks::Chunk::append()
@ 0x32acf5f starrocks::ChunksSorterFullSort::_partial_sort()
@ 0x32ad441 starrocks::ChunksSorterFullSort::update()
@ 0x3410026 starrocks::pipeline::PartitionSortSinkOperator::push_chunk()
@ 0x310fef9 starrocks::pipeline::PipelineDriver::process()
@ 0x578d84b starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x50e5222 starrocks::ThreadPool::dispatch_thread()
@ 0x50dfd1a starrocks::thread::supervise_thread()
@ 0x7f9a36628ea5 start_thread
@ 0x7f9a35c439fd __clone
@ 0x0 (unknown)
start time: Thu Sep 21 10:47:20 CST 2023
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/starrocks/be/lib/jni-packages/starrocks-jdbc-bridge-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/starrocks/be/lib/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

用了窗口函数?这个query id对应的query看下,获取下explain costs + sql

这个queryid我看了是查询的表结构
/* ApplicationName=DataGrip 2023.1.2 */ select ordinal_position, column_name, column_type, column_default, null as generation_expression, table_name, column_comment, is_nullable, extra, collation_name from information_schema.columns where table_schema = ‘ods’ order by table_name, ordinal_position

如果能稳定复现,麻烦获取下coredump 如何获取coredump