版本2.5.2
从2.3.2->2.4.4->2.5.2一路升级上来，3节点，升级完后3小时内其中两个BE挂掉3次；
BE上做了内存限制，下面是BE的配置参数：
query_mem_limit = 12884901888
#mem_limit = 60%
load_process_max_memory_limit_percent = 60
big_query_mem_limit = 5242880
# too_many_version
cumulative_compaction_num_threads_per_disk = 4
base_compaction_num_threads_per_disk = 4
cumulative_compaction_check_interval_seconds = 2
base_compaction_check_interval_seconds = 5
下面是BE报错信息：
宕机节点一报错：
I0303 09:31:36.483415 503624 txn_manager.cpp:213] Commit txn successfully. tablet: 1514017, txn_id: 1114730, rowsetid: 02000000000327029b4bafe43efc9eaf0faa1241e9251b86 #segment:0 #delfile:0
I0303 09:31:36.487126 508781 pipeline_driver_executor.cpp:224] [Driver] Succeed to report exec state: fragment_instance_id=23588ca6-b963-11ed-9f82-fa163e777789
I0303 09:31:36.507130 503268 local_tablets_channel.cpp:483] LocalTabletsChannel txn_id: 1114731 load_id: 764b0b9a-c752-cc9c-6526-89499b9cafb4 open delta writer: [1978740:0][1978764:0][1978752:0][1978776:0][1978812:0][1978800:0][1978704:0][1978824:0][1978716:0][1978744:0][1999925:0][1978772:0][1978792:0][1978760:0][1978796:0][1978820:0][1978808:0][1978832:0][1978724:0][1978840:0][1978869:0][1992886:0][1978844:0][2018435:0][1978728:0][1978736:0][1978784:0][1978732:0][1978720:0][1978836:0][1978712:0][1978828:0][1978816:0][1978788:0][2031357:0][1978780:0][1978804:0][1978768:0][1978748:0][1978874:0][1978756:0][1978708:0] failed_tablets:
W0303 09:31:36.639590 503369 stack_util.cpp:126] 2023-03-03 09:31:36.639556, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
@ 0x2a19ca8 _Znwm.cold
@ 0x45807c4 starrocks::vectorized::ReplaceAggregator<>::append_data()
@ 0x458beb2 starrocks::vectorized::ValueColumnAggregator<>::aggregate_values()
@ 0x408b55e starrocks::vectorized::ChunkAggregator::aggregate()
@ 0x408c078 starrocks::vectorized::MemTable::_aggregate()
@ 0x408ed39 starrocks::vectorized::MemTable::_merge()
@ 0x408f2d0 starrocks::vectorized::MemTable::finalize()
@ 0x4590022 starrocks::vectorized::DeltaWriter::_flush_memtable_async()
@ 0x4591195 starrocks::vectorized::DeltaWriter::close()
@ 0x5000728 starrocks::vectorized::AsyncDeltaWriter::_execute()
@ 0x586930c bthread::ExecutionQueueBase::_execute()
@ 0x586a088 bthread::ExecutionQueueBase::_execute_tasks()
@ 0x47d2acd starrocks::ThreadPool::dispatch_thread()
@ 0x47cd85a starrocks::Thread::supervise_thread()
@ 0x7f5883327ea5 start_thread
@ 0x7f58829429fd __clone
@ (nil) (unknown)
W0303 09:31:36.708330 503268 stack_util.cpp:126] 2023-03-03 09:31:36.708277, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
@ 0x2a19ca8 _Znwm.cold
@ 0x7c6250c std::__cxx11::basic_string<>::reserve()
@ 0x5a93540 google::protobuf::internal::EpsCopyInputStream::ReadStringFallback()
@ 0x4aed747 starrocks::ChunkPB::_InternalParse()
@ 0x4b2c0f5 google::protobuf::internal::ParseContext::ParseMessage<>()
@ 0x4b1def5 starrocks::PTabletWriterAddChunkRequest::_InternalParse()
@ 0x5a904db google::protobuf::MessageLite::MergeFromImpl()
@ 0x58e2336 brpc::ParsePbFromIOBuf()
@ 0x59706ee brpc::policy::ProcessRpcRequest()
@ 0x58b2677 brpc::ProcessInputMessage()
@ 0x58b354b brpc::InputMessenger::OnNewMessages()
@ 0x58a3ade brpc::Socket::ProcessEvent()
@ 0x587682f bthread::TaskGroup::task_runner()
@ 0x59be181 bthread_make_fcontext
I0303 09:31:36.734573 502797 daemon.cpp:188] Current memory statistics: process(13865001720), query_pool(105056526), load(28497488), metadata(121266574), compaction(20648), schema_change(0), column_pool(1224715502), page_cache(9007626416), update(2272129976), chunk_allocator(431210240), clone(0), consistency(0)
W0303 09:31:36.787966 503282 stack_util.cpp:126] 2023-03-03 09:31:36.787923, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
@ 0x2a19ca8 _Znwm.cold
@ 0x2b8e084 starrocks::vectorized::BinaryColumnBase<>::reserve()
@ 0x4364de4 starrocks::vectorized::CompactionState::_load_segments()
@ 0x4365ecb starrocks::vectorized::CompactionState::_do_load()
@ 0x4365f75 _ZZSt9call_onceIZN9starrocks10vectorized15CompactionState4loadEPNS0_6RowsetEEUlvE_JEEvRSt9once_flagOT_DpOT0_ENUlvE0_4_FUNEv
@ 0x7f588332620b __pthread_once_slow
@ 0x4364518 starrocks::vectorized::CompactionState::load()
@ 0x3f5f737 starrocks::TabletUpdates::_commit_compaction()
@ 0x3f68f50 starrocks::TabletUpdates::_do_compaction()
@ 0x3f6a13c starrocks::TabletUpdates::compaction()
@ 0x3ecf763 starrocks::StorageEngine::_perform_update_compaction()
@ 0x411d43e starrocks::StorageEngine::_update_compaction_thread_callback()
@ 0x7c64900 execute_native_thread_routine
@ 0x7f5883327ea5 start_thread
@ 0x7f58829429fd __clone
@ (nil) (unknown)
I0303 09:31:36.942579 503314 load_channel_mgr.cpp:225] Memory consumption(bytes) limit=21823517933 current=28497488 peak=325118656
宕机节点二报错：
I0303 09:34:16.134061 249186 profile_report_worker.cpp:11] register_non_pipeline_load fragment_instance_id=1f4a4dd0-055a-1d5f-1b9c-b99cdb0b3bbe
I0303 09:34:16.135140 249458 local_tablets_channel.cpp:483] LocalTabletsChannel txn_id: 1115814 load_id: 1f4a4dd0-055a-1d5f-1b9c-b99cdb0b3bbd open delta writer: [1980345:0][1980369:0][1980357:0][1980393:0][1980381:0][1980413:0][1980405:0][1980297:0][1980289:0][1999852:0][1980337:0][1980305:0][1980317:0][1980321:0][1980353:0][1980341:0][1980365:0][1980401:0][1980285:0][1980389:0][1980281:0][1980426:0][1980397:0][1980377:0][1980325:0][1980293:0][1980409:0][1980333:0][2018388:0][1980373:0][1980329:0][2031305:0][1992836:0][1980431:0][1980313:0][1980301:0][1980417:0][1980309:0][1980421:0][1980385:0][1980349:0][1980361:0] failed_tablets:
I0303 09:34:16.136667 249459 local_tablets_channel.cpp:483] LocalTabletsChannel txn_id: 1115815 load_id: 884a9dfb-7825-4356-12ec-0d26cb4db9be open delta writer: [1980239:0][1980235:0][1980231:0][1980227:0][1980223:0][1980219:0] failed_tablets:
I0303 09:34:16.186249 249459 local_tablets_channel.cpp:343] LocalTabletsChannel txn_id: 1115815 load_id: 884a9dfb-7825-4356-12ec-0d26cb4db9be commit 6 tablets: 1980239,1980235,1980231,1980227,1980223,1980219
I0303 09:34:16.190435 249879 txn_manager.cpp:213] Commit txn successfully. tablet: 1980219, txn_id: 1115815, rowsetid: 020000000003aaf347463702deff76095ea42a6eeb1207b4 #segment:1 #delfile:0
I0303 09:34:16.190759 249063 txn_manager.cpp:213] Commit txn successfully. tablet: 1980223, txn_id: 1115815, rowsetid: 020000000003aaf447463702deff76095ea42a6eeb1207b4 #segment:1 #delfile:0
I0303 09:34:16.193754 249060 txn_manager.cpp:213] Commit txn successfully. tablet: 1980239, txn_id: 1115815, rowsetid: 020000000003aaf847463702deff76095ea42a6eeb1207b4 #segment:1 #delfile:0
I0303 09:34:16.194216 249067 txn_manager.cpp:213] Commit txn successfully. tablet: 1980235, txn_id: 1115815, rowsetid: 020000000003aaf747463702deff76095ea42a6eeb1207b4 #segment:1 #delfile:0
I0303 09:34:16.196628 250090 txn_manager.cpp:213] Commit txn successfully. tablet: 1980227, txn_id: 1115815, rowsetid: 020000000003aaf547463702deff76095ea42a6eeb1207b4 #segment:1 #delfile:0
I0303 09:34:16.196931 250091 txn_manager.cpp:213] Commit txn successfully. tablet: 1980231, txn_id: 1115815, rowsetid: 020000000003aaf647463702deff76095ea42a6eeb1207b4 #segment:1 #delfile:0
I0303 09:34:16.228986 249567 task_worker_pool.cpp:158] Submit task success. type=PUBLISH_VERSION, signature=1115815, task_count_in_queue=1
I0303 09:34:16.229014 249440 task_worker_pool.cpp:490] get publish version task txn_id: 1115815 priority queue size: 1
I0303 09:34:16.229238 249409 tablet_updates.cpp:512] commit rowset tablet:1980223 version:29070 txn_id: 1115815 020000000003aaf447463702deff76095ea42a6eeb1207b4 rowset:32508 #seg:1 #delfile:0 #row:808 size:48.36 KB #pending:0
I0303 09:34:16.229274 249409 publish_version.cpp:110] Publish txn success tablet:1980223 version:29070 tablet_max_version:29070 partition:1980216 txn_id: 1115815 rowset:020000000003aaf447463702deff76095ea42a6eeb1207b4
I0303 09:34:16.229362 249406 tablet_updates.cpp:512] commit rowset tablet:1980227 version:29070 txn_id: 1115815 020000000003aaf547463702deff76095ea42a6eeb1207b4 rowset:32486 #seg:1 #delfile:0 #row:872 size:51.93 KB #pending:0
I0303 09:34:16.229362 249407 tablet_updates.cpp:512] commit rowset tablet:1980235 version:29070 txn_id: 1115815 020000000003aaf747463702deff76095ea42a6eeb1207b4 rowset:32512 #seg:1 #delfile:0 #row:866 size:50.14 KB #pending:0
I0303 09:34:16.229382 249411 tablet_updates.cpp:512] commit rowset tablet:1980219 version:29070 txn_id: 1115815 020000000003aaf347463702deff76095ea42a6eeb1207b4 rowset:32465 #seg:1 #delfile:0 #row:881 size:52.35 KB #pending:0
I0303 09:34:16.229395 249410 tablet_updates.cpp:512] commit rowset tablet:1980239 version:29070 txn_id: 1115815 020000000003aaf847463702deff76095ea42a6eeb1207b4 rowset:32448 #seg:1 #delfile:0 #row:854 size:50.48 KB #pending:0
I0303 09:34:16.229365 249408 tablet_updates.cpp:512] commit rowset tablet:1980231 version:29070 txn_id: 1115815 020000000003aaf647463702deff76095ea42a6eeb1207b4 rowset:32510 #seg:1 #delfile:0 #row:915 size:53.13 KB #pending:0
I0303 09:34:16.229439 249406 publish_version.cpp:110] Publish txn success tablet:1980227 version:29070 tablet_max_version:29070 partition:1980216 txn_id: 1115815 rowset:020000000003aaf547463702deff76095ea42a6eeb1207b4
I0303 09:34:16.229442 249407 publish_version.cpp:110] Publish txn success tablet:1980235 version:29070 tablet_max_version:29070 partition:1980216 txn_id: 1115815 rowset:020000000003aaf747463702deff76095ea42a6eeb1207b4
E0303 09:34:16.229431 249408 threadpool.cpp:433] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
I0303 09:34:16.229457 249411 publish_version.cpp:110] Publish txn success tablet:1980219 version:29070 tablet_max_version:29070 partition:1980216 txn_id: 1115815 rowset:020000000003aaf347463702deff76095ea42a6eeb1207b4
I0303 09:34:16.229522 249408 publish_version.cpp:110] Publish txn success tablet:1980231 version:29070 tablet_max_version:29070 partition:1980216 txn_id: 1115815 rowset:020000000003aaf647463702deff76095ea42a6eeb1207b4
E0303 09:34:16.229533 249410 threadpool.cpp:433] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
I0303 09:34:16.229552 249410 publish_version.cpp:110] Publish txn success tablet:1980239 version:29070 tablet_max_version:29070 partition:1980216 txn_id: 1115815 rowset:020000000003aaf847463702deff76095ea42a6eeb1207b4
I0303 09:34:16.229584 249440 publish_version.cpp:185] publish_version success. txn_id: 1115815 #partition:1 #tablet:6 time:0ms #already_finished:0
I0303 09:34:16.230026 256244 tablet_updates.cpp:1051] apply_rowset_commit finish. tablet:1980223 version:29070 txn_id: 1115815 total del/row:79449/937081 8% rowset:32508 #seg:1 #op(upsert:808 del:0) #del:78641+808=79449 #dv:2 duration:1ms(0/0/1/0)
I0303 09:34:16.230232 256246 tablet_updates.cpp:1051] apply_rowset_commit finish. tablet:1980227 version:29070 txn_id: 1115815 total del/row:79569/938184 8% rowset:32486 #seg:1 #op(upsert:872 del:0) #del:78697+872=79569 #dv:2 duration:1ms(0/1/0/0)
I0303 09:34:16.230473 256247 tablet_updates.cpp:1051] apply_rowset_commit finish. tablet:1980219 version:29070 txn_id: 1115815 total del/row:64083/921102 6% rowset:32465 #seg:1 #op(upsert:881 del:0) #del:63202+881=64083 #dv:2 duration:1ms(0/1/0/0)
I0303 09:34:16.230675 256248 tablet_updates.cpp:1051] apply_rowset_commit finish. tablet:1980235 version:29070 txn_id: 1115815 total del/row:62103/919388 6% rowset:32512 #seg:1 #op(upsert:866 del:0) #del:61237+866=62103 #dv:2 duration:1ms(0/1/0/0)
I0303 09:34:16.230705 256244 tablet_updates.cpp:1051] apply_rowset_commit finish. tablet:1980239 version:29070 txn_id: 1115815 total del/row:63733/920701 6% rowset:32448 #seg:1 #op(upsert:854 del:0) #del:62879+854=63733 #dv:2 duration:0ms(0/0/0/0)
I0303 09:34:16.230731 256246 tablet_updates.cpp:1051] apply_rowset_commit finish. tablet:1980231 version:29070 txn_id: 1115815 total del/row:64001/922152 6% rowset:32510 #seg:1 #op(upsert:915 del:0) #del:63086+915=64001 #dv:2 duration:0ms(0/0/0/0)
I0303 09:34:16.232626 249440 task_worker_pool.cpp:512] batch flush 1 txn publish task(s). #dir:1 flush:3ms finish_task_rpc:0ms
W0303 09:34:16.454488 249547 stack_util.cpp:126] 2023-03-03 09:34:16.454452, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
@ 0x2a19ca8 _Znwm.cold
@ 0x4c916b8 starrocks::ByteBuffer::allocate()
@ 0x4c9105d starrocks::StreamLoadAction::on_chunk_data()
@ 0x57aaf28 evhttp_read_body
@ 0x57ae603 bufferevent_readcb
@ 0x579ad42 event_process_active_single_queue
@ 0x579b47f event_base_loop
@ 0x4c73f14 _ZZN9starrocks12EvHttpServer5startEvENKUlvE_clEv
@ 0x7c64900 execute_native_thread_routine
@ 0x7f42ea91bea5 start_thread
@ 0x7f42e9f369fd __clone
@ (nil) (unknown)
W0303 09:34:16.471014 252918 stack_util.cpp:126] 2023-03-03 09:34:16.470986, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
@ 0x2a19ca8 _Znwm.cold
@ 0x2b9fd10 std::vector<>::_M_default_append()
@ 0x2c20e2f starrocks::vectorized::BinaryColumnBase<>::append_selective()
@ 0x408fe7a starrocks::vectorized::MemTable::insert()
@ 0x4590913 starrocks::vectorized::DeltaWriter::write()
@ 0x5000697 starrocks::vectorized::AsyncDeltaWriter::_execute()
@ 0x586930c bthread::ExecutionQueueBase::_execute()
@ 0x586a088 bthread::ExecutionQueueBase::_execute_tasks()
@ 0x47d2acd starrocks::ThreadPool::dispatch_thread()
@ 0x47cd85a starrocks::Thread::supervise_thread()
@ 0x7f42ea91bea5 start_thread
@ 0x7f42e9f369fd __clone
@ (nil) (unknown)
I0303 09:34:16.486944 249514 local_tablets_channel.cpp:483] LocalTabletsChannel txn_id: 1115816 load_id: 6048248d-ec6e-9d94-e7a4-4efa2e75d384 open delta writer: [1998706:0][1998702:0][1998726:0][1998714:0][1998750:0][1998774:0][1998762:0][1998666:0][1998786:0][1998678:0][1998718:0][2031300:0][1998794:0][1998686:0][1998806:0][1998690:0][1998738:0][1998856:0][1998682:0][1998798:0][1998698:0][1998770:0][1998802:0][1998861:0][2018383:0][1998866:0][1998670:0][1999968:0][1998694:0][1998782:0][1998746:0][1998758:0][1998722:0][1998734:0][1998674:0][1998790:0][1998778:0][1998754:0][1998766:0][1998730:0][1998742:0][1998710:0] failed_tablets:
I0303 09:34:16.503283 249561 stream_load.cpp:210] new income streaming load request.id=a2425d5d5c20ebfd-5629d94a39f5a69c, job_id=-1, txn_id: -1, label=925613f3-656b-49d3-a681-29347d6f9475, db=crb_ods_ocms, db=crb_ods_ocms, tbl=sl_receiving_bill_detailed
I0303 09:34:16.506538 249561 stream_load_executor.cpp:55] begin to execute job. label=925613f3-656b-49d3-a681-29347d6f9475, txn_id: 1115817, query_id=a2425d5d-5c20-ebfd-5629-d94a39f5a69c
I0303 09:34:16.506577 249561 plan_fragment_executor.cpp:70] Prepare(): query_id=a2425d5d-5c20-ebfd-5629-d94a39f5a69c fragment_instance_id=a2425d5d-5c20-ebfd-5629-d94a39f5a69d backend_num=0
I0303 09:34:16.507450 249189 plan_fragment_executor.cpp:176] Open(): fragment_instance_id=a2425d5d-5c20-ebfd-5629-d94a39f5a69d
I0303 09:34:16.507469 249189 profile_report_worker.cpp:11] register_non_pipeline_load fragment_instance_id=a2425d5d-5c20-ebfd-5629-d94a39f5a69d
I0303 09:34:16.508273 249464 local_tablets_channel.cpp:483] LocalTabletsChannel txn_id: 1115817 load_id: a2425d5d-5c20-ebfd-5629-d94a39f5a69c open delta writer: [1998826:0][1998524:0][1998520:0][1998544:0][1998532:0][1998568:0][1998556:0][1998592:0][1998580:0][1998500:0][1998616:0][1998552:0][1998540:0][1998576:0][1998836:0][1998600:0][1998484:0][1998588:0][1998612:0][1998496:0][1998504:0][1998620:0][1998624:0][1998508:0][1998516:0][1998564:0][1998512:0][1998492:0][1998608:0][1998596:0][1998572:0][1998584:0][1999905:0][2031352:0][1998548:0][1998560:0][1998831:0][1998536:0][1998528:0][2018430:0][1998488:0][1998604:0] failed_tablets:
I0303 09:34:16.590667 249459 local_tablets_channel.cpp:343] LocalTabletsChannel txn_id: 1115816 load_id: 6048248d-ec6e-9d94-e7a4-4efa2e75d384 commit 1 tablets: 1998861
I0303 09:34:16.716156 249456 local_tablets_channel.cpp:483] LocalTabletsChannel txn_id: 1115818 load_id: c442f46e-ef5c-d284-7c9a-4de7f88efdac open delta writer: [2021805:0][2021801:0][2021797:0][2021793:0][2021785:0][2021781:0][2021777:0][2021773:0][2021769:0][2021765:0][2021761:0][2021789:0][2031347:0][2021693:0][2021689:0][2021685:0][2021681:0][2021677:0][2021673:0][2021669:0][2021665:0][2021661:0][2021657:0][2021653:0][2021649:0][2021645:0][2021697:0][2021701:0][2021705:0][2021709:0][2021713:0][2021717:0][2021721:0][2021725:0][2021729:0][2021733:0][2021737:0][2021741:0][2021745:0][2021749:0][2021753:0][2021757:0] failed_tablets:
W0303 09:34:16.790210 249557 stack_util.cpp:126] 2023-03-03 09:34:16.790176, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
@ 0x2a19ca8 _Znwm.cold
@ 0x4c916b8 starrocks::ByteBuffer::allocate()
@ 0x4c9105d starrocks::StreamLoadAction::on_chunk_data()
@ 0x57aaf28 evhttp_read_body
@ 0x57ae603 bufferevent_readcb
@ 0x579ad42 event_process_active_single_queue
@ 0x579b47f event_base_loop
@ 0x4c73f14 _ZZN9starrocks12EvHttpServer5startEvENKUlvE_clEv
@ 0x7c64900 execute_native_thread_routine
@ 0x7f42ea91bea5 start_thread
@ 0x7f42e9f369fd __clone
@ (nil) (unknown)
I0303 09:34:16.835374 249562 stream_load.cpp:210] new income streaming load request.id=2e4e6f702225f7d9-c145c09407c99fb2, job_id=-1, txn_id: -1, label=dfc84e25-b86f-4ef6-96e6-47ac1d3ab68c, db=crb_ods_ocms, db=crb_ods_ocms, tbl=order_status
I0303 09:34:16.838981 249562 stream_load_executor.cpp:55] begin to execute job. label=dfc84e25-b86f-4ef6-96e6-47ac1d3ab68c, txn_id: 1115819, query_id=2e4e6f70-2225-f7d9-c145-c09407c99fb2
I0303 09:34:16.839016 249562 plan_fragment_executor.cpp:70] Prepare(): query_id=2e4e6f70-2225-f7d9-c145-c09407c99fb2 fragment_instance_id=2e4e6f70-2225-f7d9-c145-c09407c99fb3 backend_num=0
I0303 09:34:16.839905 249185 plan_fragment_executor.cpp:176] Open(): fragment_instance_id=2e4e6f70-2225-f7d9-c145-c09407c99fb3
I0303 09:34:16.839931 249185 profile_report_worker.cpp:11] register_non_pipeline_load fragment_instance_id=2e4e6f70-2225-f7d9-c145-c09407c99fb3
I0303 09:34:16.840950 249514 local_tablets_channel.cpp:483] LocalTabletsChannel txn_id: 1115819 load_id: 2e4e6f70-2225-f7d9-c145-c09407c99fb2 open delta writer: [1979330:0][1979326:0][1979350:0][2018337:0][1979338:0][1979374:0][1979362:0][1979398:0][1979386:0][1979290:0][1979410:0][1979302:0][1979382:0][1979406:0][1979394:0][1980496:0][1979418:0][1979402:0][1979310:0][1979426:0][1979430:0][1979314:0][1979322:0][1979370:0][1992891:0][1980501:0][1979318:0][1979346:0][1979358:0][1979306:0][1979422:0][1979298:0][1979414:0][1979378:0][1979390:0][1979354:0][1979366:0][2031254:0][1979334:0][1979342:0][1979294:0][1999935:0] failed_tablets:
W0303 09:34:17.226037 249561 stack_util.cpp:126] 2023-03-03 09:34:17.226004, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
@ 0x2a19ca8 _Znwm.cold
@ 0x4c916b8 starrocks::ByteBuffer::allocate()
@ 0x4c9105d starrocks::StreamLoadAction::on_chunk_data()
@ 0x57aaf28 evhttp_read_body
@ 0x57ae603 bufferevent_readcb
@ 0x579ad42 event_process_active_single_queue
@ 0x579b47f event_base_loop
@ 0x4c73f14 _ZZN9starrocks12EvHttpServer5startEvENKUlvE_clEv
@ 0x7c64900 execute_native_thread_routine
@ 0x7f42ea91bea5 start_thread
@ 0x7f42e9f369fd __clone
@ (nil) (unknown)
I0303 09:34:17.264967 249464 local_tablets_channel.cpp:483] LocalTabletsChannel txn_id: 1115820 load_id: 1f475481-b010-690b-e3e0-eb6b6fb7f3a3 open delta writer: [1980239:0][1980235:0][1980231:0][1980227:0][1980223:0][1980219:0] failed_tablets:
I0303 09:34:17.350672 249571 heartbeat_server.cpp:80] Updating master info: TMasterInfo(network_address=TNetworkAddress(hostname=10.207.66.39, port=9020), cluster_id=1694552347, epoch=53, token=53391ffe-09fe-4279-b7b5-9e591860eb4e, backend_ip=10.207.66.67, http_port=8030, heartbeat_flags=0, backend_id=10014, min_active_txn_id=1115809)
W0303 09:34:17.396297 250091 stack_util.cpp:126] 2023-03-03 09:34:17.396264, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
@ 0x2a19ca8 _Znwm.cold
@ 0x2c0ab9b std::vector<>::_M_default_append()
@ 0x2c0b06c starrocks::vectorized::FixedLengthColumnBase<>::append_selective()
@ 0x408b5a1 starrocks::vectorized::ChunkAggregator::aggregate()
@ 0x408c078 starrocks::vectorized::MemTable::_aggregate()
@ 0x408ed39 starrocks::vectorized::MemTable::_merge()
@ 0x408f2d0 starrocks::vectorized::MemTable::finalize()
@ 0x4590022 starrocks::vectorized::DeltaWriter::_flush_memtable_async()
@ 0x4591195 starrocks::vectorized::DeltaWriter::close()
@ 0x5000728 starrocks::vectorized::AsyncDeltaWriter::_execute()
@ 0x586930c bthread::ExecutionQueueBase::_execute()
@ 0x586a088 bthread::ExecutionQueueBase::_execute_tasks()
@ 0x47d2acd starrocks::ThreadPool::dispatch_thread()
@ 0x47cd85a starrocks::Thread::supervise_thread()
@ 0x7f42ea91bea5 start_thread
@ 0x7f42e9f369fd __clone
@ (nil) (unknown)
W0303 09:34:17.548039 249562 stack_util.cpp:126] 2023-03-03 09:34:17.548008, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
@ 0x2a19ca8 _Znwm.cold
@ 0x4c916b8 starrocks::ByteBuffer::allocate()
@ 0x4c9105d starrocks::StreamLoadAction::on_chunk_data()
@ 0x57aaf28 evhttp_read_body
@ 0x57ae603 bufferevent_readcb
@ 0x579ad42 event_process_active_single_queue
@ 0x579b47f event_base_loop
@ 0x4c73f14 _ZZN9starrocks12EvHttpServer5startEvENKUlvE_clEv
@ 0x7c64900 execute_native_thread_routine
@ 0x7f42ea91bea5 start_thread
@ 0x7f42e9f369fd __clone
@ (nil) (unknown)
W0303 09:34:17.986771 249514 stack_util.cpp:126] 2023-03-03 09:34:17.986744, query_id=00000000-0000-0000-0000-000000000000, fragment_instance_id=00000000-0000-0000-0000-000000000000 throws exception: std::bad_alloc, trace:
@ 0x2a19ca8 _Znwm.cold
@ 0x7c6250c std::__cxx11::basic_string<>::reserve()
@ 0x5a93540 google::protobuf::internal::EpsCopyInputStream::ReadStringFallback()
@ 0x4aed747 starrocks::ChunkPB::_InternalParse()
@ 0x4b2c0f5 google::protobuf::internal::ParseContext::ParseMessage<>()
@ 0x4b1def5 starrocks::PTabletWriterAddChunkRequest::_InternalParse()
@ 0x5a904db google::protobuf::MessageLite::MergeFromImpl()
@ 0x58e2336 brpc::ParsePbFromIOBuf()
@ 0x59706ee brpc::policy::ProcessRpcRequest()
@ 0x58b2677 brpc::ProcessInputMessage()
@ 0x58b354b brpc::InputMessenger::OnNewMessages()
@ 0x58a3ade brpc::Socket::ProcessEvent()
@ 0x587682f bthread::TaskGroup::task_runner()
@ 0x59be181 bthread_make_fcontext
