为了更快的定位您的问题，请提供以下信息，谢谢
【详述】问题详细描述
日志显示11点35分创建线程失败, 11点39分be自动重启.
监控显示系统盘ioutils特别高, 线程上下文切换也特别高, 数据盘目录在另外一个磁盘
be.out:
F20250624 11:33:50.921194 139715813529152 tablet_updates.cpp:930] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:310943770 #version:153 [71278 71425@151 71426] pending: rowsets:13
71768 [seg:1 row:129 del:0 bytes:29207 row_size:0 compaction_score:268406249 compaction_level:-1 partial_update_by_column:false]
71998 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
71999 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72000 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72001 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72002 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72003 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72004 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72005 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72006 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72007 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72008 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72009 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
branch-3.3.12 RELEASE (build 9a77cc1)
query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000
tracker:process consumption: 23745297648
tracker:jemalloc_metadata consumption: 1169797360
tracker:query_pool consumption: 4939276320
tracker:query_pool/connector_scan consumption: 0
tracker:load consumption: 32664
tracker:metadata consumption: 9204638969
tracker:tablet_metadata consumption: 1039640775
tracker:rowset_metadata consumption: 351074658
tracker:segment_metadata consumption: 1106512109
tracker:column_metadata consumption: 6707411427
F20250624 11:33:50.921646 139627530729024 tablet_updates.cpp:930] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:310943774 #version:153 [71278 71425@151 71426] pending: rowsets:12
72000 [seg:1 row:149 del:0 bytes:32116 row_size:0 compaction_score:268403340 compaction_level:-1 partial_update_by_column:false]
72001 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72002 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72003 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72004 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72005 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72006 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72007 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72008 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72009 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72010 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72011 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
tracker:tablet_schema consumption: 3535935
tracker:segment_zonemap consumption: 1044694486
tracker:short_key_index consumption: 2709364
tracker:column_zonemap_index consumption: 2229519331
tracker:ordinal_index consumption: 1444412640
tracker:bitmap_index consumption: 0
branch-3.3.12 RELEASE (build 9a77cc1)
tracker:bloom_filter_index consumption: 0
be.WARN:
W20250624 11:25:06.077151 139715477300800 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=020000001a78e279764ea9eaba931dc98ad7f0669add3d9f tablet: 352022990
W20250624 11:25:07.443548 139715477300800 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=020000001a79d854764ea9eaba931dc98ad7f0669add3d9f tablet: 352042525
W20250624 11:25:09.051062 139715477300800 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=020000001a79d853764ea9eaba931dc98ad7f0669add3d9f tablet: 352042519
W20250624 11:25:09.896402 139715477300800 storage_engine.cpp:1144] traverse_rowset_meta and remove 5/14433 invalid rowset metas, path:/dbdata1 duration:4618ms
W20250624 11:25:16.807346 139715485693504 rowset.cpp:331] Fail to delete /dbdata1/data/202/352022990/1369509696/020000001a78e279764ea9eaba931dc98ad7f0669add3d9f_0.dat: Not found: /dbdata1/data/202/352022990/1369509696/020000001a78e279764ea9eaba931dc98ad7f0669add3d9f_0.dat: No such file or directory
W20250624 11:25:16.807381 139715485693504 rowset.cpp:331] Fail to delete /dbdata1/data/202/352022990/1369509696/020000001a78e279764ea9eaba931dc98ad7f0669add3d9f_1.dat: Not found: /dbdata1/data/202/352022990/1369509696/020000001a78e279764ea9eaba931dc98ad7f0669add3d9f_1.dat: No such file or directory
W20250624 11:25:16.807391 139715485693504 rowset.cpp:331] Fail to delete /dbdata1/data/202/352022990/1369509696/020000001a78e279764ea9eaba931dc98ad7f0669add3d9f_2.dat: Not found: /dbdata1/data/202/352022990/1369509696/020000001a78e279764ea9eaba931dc98ad7f0669add3d9f_2.dat: No such file or directory
W20250624 11:25:16.807400 139715485693504 rowset.cpp:331] Fail to delete /dbdata1/data/202/352022990/1369509696/020000001a78e279764ea9eaba931dc98ad7f0669add3d9f_3.dat: Not found: /dbdata1/data/202/352022990/1369509696/020000001a78e279764ea9eaba931dc98ad7f0669add3d9f_3.dat: No such file or directory
W20250624 11:28:25.806129 139715477300800 storage_engine.cpp:1144] traverse_rowset_meta and remove 104/10439 invalid rowset metas, path:/dbdata1 duration:5802ms
E20250624 11:29:29.954007 139716634883648 scan_operator.cpp:465] scan fragment 70130612-50ab-11f0-9020-024275dda844 driver 0 Scan tasks error: Cancelled: canceled state
W20250624 11:31:36.140605 139715477300800 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=020000001a7b1572764ea9eaba931dc98ad7f0669add3d9f tablet: 352063822
W20250624 11:31:37.779799 139715477300800 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=020000001a7b1576764ea9eaba931dc98ad7f0669add3d9f tablet: 352063852
W20250624 11:31:38.600520 139715477300800 storage_engine.cpp:1144] traverse_rowset_meta and remove 2/12422 invalid rowset metas, path:/dbdata1 duration:4661ms
F20250624 11:33:50.921194 139715813529152 tablet_updates.cpp:930] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:310943770 #version:153 [71278 71425@151 71426] pending: rowsets:13
71768 [seg:1 row:129 del:0 bytes:29207 row_size:0 compaction_score:268406249 compaction_level:-1 partial_update_by_column:false]
71998 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
71999 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72000 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72001 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72002 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72003 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72004 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72005 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72006 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72007 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72008 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72009 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
F20250624 11:33:50.921646 139627530729024 tablet_updates.cpp:930] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:310943774 #version:153 [71278 71425@151 71426] pending: rowsets:12
72000 [seg:1 row:149 del:0 bytes:32116 row_size:0 compaction_score:268403340 compaction_level:-1 partial_update_by_column:false]
72001 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72002 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72003 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72004 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72005 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72006 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72007 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72008 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72009 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72010 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72011 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
F20250624 11:33:50.921646 139627530729024 tablet_updates.cpp:930] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:310943774 #version:153 [71278 71425@151 71426] pending: rowsets:12
72000 [seg:1 row:149 del:0 bytes:32116 row_size:0 compaction_score:268403340 compaction_level:-1 partial_update_by_column:false]
72001 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72002 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72003 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72004 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72005 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72006 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72007 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72008 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72009 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72010 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72011 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]F20250624 11:34:19.136047 139715393373760 tablet_updates.cpp:930] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:151284448 #version:55 [53152.1 53204@53 53204.1] pending: rowsets:
1
54468 [seg:1 row:3271 del:0 bytes:76268 row_size:0 compaction_score:268359188 compaction_level:-1 partial_update_by_column:false]
E20250624 11:33:50.901830 139715779958336 threadpool.cpp:468] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
@ 0x6326535 starrocks::get_stack_trace[abi:cxx11]()
@ 0x86ff741 starrocks::ThreadPool::do_submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPoolToken*, starrocks::ThreadPool::Priority)
@ 0x86ffc22 starrocks::ThreadPoolToken::submit_func(std::function<void ()>, starrocks::ThreadPool::Priority)
@ 0x8428926 starrocks::run_publish_version_task(starrocks::ThreadPoolToken*, starrocks::TPublishVersionRequest const&, starrocks::TFinishTaskRequest&, std::unordered_set<starrocks::DataDir*, std::hash<starrocks::DataDir*>, std::equal_to<starrocks::DataDir*>, std::allo^W^L
@ 0x8416f98 starrocks::PublishVersionTaskWorkerPool::_worker_thread_callback(void*)
@ 0xe48f0c4 execute_native_thread_routine
@ 0x7f1369498ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f1369529a04 clone
E20250624 11:33:51.110592 139715426944576 threadpool.cpp:468] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
@ 0x6326535 starrocks::get_stack_trace[abi:cxx11]()
@ 0x86ff741 starrocks::ThreadPool::do_submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPoolToken*, starrocks::ThreadPool::Priority)
@ 0x86ffdc5 starrocks::ThreadPool::submit_func(std::function<void ()>, starrocks::ThreadPool::Priority)
@ 0x6854b1f starrocks::CompactionManager::_dispatch_worker()
@ 0xe48f0c4 execute_native_thread_routine
@ 0x7f1369498ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f1369529a04 clone
E20250624 11:34:06.827800 139700631234112 threadpool.cpp:468] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
@ 0x6326535 starrocks::get_stack_trace[abi:cxx11]()
@ 0x86ff741 starrocks::ThreadPool::do_submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPoolToken*, starrocks::ThreadPool::Priority)
@ 0x86ffdc5 starrocks::ThreadPool::submit_func(std::function<void ()>, starrocks::ThreadPool::Priority)
@ 0x8405c60 starrocks::AgentServer::Impl::submit_tasks(starrocks::TAgentResult&, std::vector<starrocks::TAgentTaskRequest, std::allocator<starrocks::TAgentTaskRequest> > const&)
@ 0x882c8d6 starrocks::BackendServiceProcessor::process_submit_tasks(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*)
@ 0x882dd30 starrocks::BackendServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, void*)
@ 0x883824b apache::thrift::TDispatchProcessor::process(std::shared_ptr<apache::thrift::protocol::TProtocol>, std::shared_ptr<apache::thrift::protocol::TProtocol>, void*)
@ 0x9a9bbf9 apache::thrift::server::TConnectedClient::run()
@ 0x9a9caf8 apache::thrift::server::TThreadedServer::TConnectedClientRunner::run()
@ 0x9a9f38f apache::thrift::concurrency::Thread::threadMain(std::shared_ptr<apache::thrift::concurrency::Thread>)
@ 0x9a76259 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(std::shared_ptr<apache::thrift::concurrency::Thread>), std::shared_ptr<apache::thrift::concurrency::Thread> > > >::_M_run()
@ 0xe48f0c4 execute_native_thread_routine
@ 0x7f1369498ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f1369529a04 clone
W20250624 11:34:34.524266 139543556630080 engine_clone_task.cpp:416] Fail to download snapshot /dbdata1/snapshot/20250624113431.10526.180/ from saas-node11:8040, status: Internal error: getaddrinfo() thread failed to start
be/src/storage/task/engine_clone_task.cpp:535 client->execute(&file_list_str)
be/src/storage/task/engine_clone_task.cpp:538 HttpClient::execute_with_retry(DOWNLOAD_FILE_MAX_RETRY, 1, list_files_cb), tablet_id:310943790, schema_hash:123018287
W20250624 11:34:34.524267 139551192376896 engine_clone_task.cpp:416] Fail to download snapshot /dbdata1/snapshot/20250624113431.10525.180/ from saas-node11:8040, status: Internal error: getaddrinfo() thread failed to start
be/src/storage/task/engine_clone_task.cpp:535 client->execute(&file_list_str)
be/src/storage/task/engine_clone_task.cpp:538 HttpClient::execute_with_retry(DOWNLOAD_FILE_MAX_RETRY, 1, list_files_cb), tablet_id:310943786, schema_hash:123018287
W20250624 11:34:34.524333 139543556630080 engine_clone_task.cpp:180] Fail to load snapshot:Internal error: getaddrinfo() thread failed to start
be/src/storage/task/engine_clone_task.cpp:535 client->execute(&file_list_str)
be/src/storage/task/engine_clone_task.cpp:538 HttpClient::execute_with_retry(DOWNLOAD_FILE_MAX_RETRY, 1, list_files_cb) tablet:310943790
W20250624 11:34:34.524344 139551192376896 engine_clone_task.cpp:180] Fail to load snapshot:Internal error: getaddrinfo() thread failed to start
be/src/storage/task/engine_clone_task.cpp:535 client->execute(&file_list_str)
be/src/storage/task/engine_clone_task.cpp:538 HttpClient::execute_with_retry(DOWNLOAD_FILE_MAX_RETRY, 1, list_files_cb) tablet:310943786
W20250624 11:34:34.524493 139543556630080 agent_task.cpp:390] clone failed. signature: 310943790
W20250624 11:34:34.525038 139551192376896 agent_task.cpp:390] clone failed. signature: 310943786
W20250624 11:34:34.849705 139716072572480 thrift_rpc_helper.cpp:129] Rpc error: FE RPC failure, address=TNetworkAddress(hostname=saas-node13, port=9020), reason=No more data to read.
W20250624 11:34:34.849782 139716408280640 thrift_rpc_helper.cpp:129] Rpc error: FE RPC failure, address=TNetworkAddress(hostname=saas-node13, port=9020), reason=No more data to read.
W20250624 11:34:35.128290 139551225935424 engine_clone_task.cpp:416] Fail to download snapshot /dbdata1/snapshot/20250624113432.10527.180/ from saas-node11:8040, status: Internal error: getaddrinfo() thread failed to start
be/src/storage/task/engine_clone_task.cpp:535 client->execute(&file_list_str)
be/src/storage/task/engine_clone_task.cpp:538 HttpClient::execute_with_retry(DOWNLOAD_FILE_MAX_RETRY, 1, list_files_cb), tablet_id:310943782, schema_hash:123018287
W20250624 11:34:35.128348 139551225935424 engine_clone_task.cpp:180] Fail to load snapshot:Internal error: getaddrinfo() thread failed to start
be/src/storage/task/engine_clone_task.cpp:535 client->execute(&file_list_str)
be/src/storage/task/engine_clone_task.cpp:538 HttpClient::execute_with_retry(DOWNLOAD_FILE_MAX_RETRY, 1, list_files_cb) tablet:310943782
E20250624 11:34:23.844944 139716676847168 threadpool.cpp:468] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
@ 0x6326535 starrocks::get_stack_trace[abi:cxx11]()
@ 0x86ff741 starrocks::ThreadPool::do_submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPoolToken*, starrocks::ThreadPool::Priority)
@ 0x86ffdc5 starrocks::ThreadPool::submit_func(std::function<void ()>, starrocks::ThreadPool::Priority)
@ 0x79d099c starrocks::pipeline::ExecStateReporter::submit(std::function<void ()>&&, bool)
@ 0x79c8d03 starrocks::pipeline::GlobalDriverExecutor::report_exec_state(starrocks::pipeline::QueryContext*, starrocks::pipeline::FragmentContext*, starrocks::Status const&, bool, bool)
@ 0x529b59e starrocks::pipeline::FragmentContext::count_down_execution_group(unsigned long)
@ 0x4fbe7ba starrocks::pipeline::PipelineDriver::finalize(starrocks::RuntimeState*, starrocks::pipeline::DriverState, long, long)
@ 0x79c7e8b starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x87032fe starrocks::ThreadPool::dispatch_thread()
@ 0x86fa569 starrocks::Thread::supervise_thread(void*)
@ 0x7f1369498ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f1369529a04 clone
F20250624 11:33:50.921646 139627530729024 tablet_updates.cpp:930] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:310943774 #version:153 [71278 71425@151 71426] pending: rowsets:12
72000 [seg:1 row:149 del:0 bytes:32116 row_size:0 compaction_score:268403340 compaction_level:-1 partial_update_by_column:false]
72001 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72002 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72003 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72004 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72005 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72006 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72007 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72008 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72009 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72010 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72011 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]F20250624 11:34:19.136047 139715393373760 tablet_updates.cpp:930] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:151284448 #version:55 [53152.1 53204@53 53204.1] pending: rowsets:
1
54468 [seg:1 row:3271 del:0 bytes:76268 row_size:0 compaction_score:268359188 compaction_level:-1 partial_update_by_column:false]F20250624 11:34:39.345424 139543598593600 tablet_updates.cpp:930] submit apply task failed: Runtime error: Could not create thread: Resource temporarily unavailable tablet:310943778 #version:156 [71275 71425@154 71426] pending: row
sets:12
71995 [seg:1 row:163 del:0 bytes:33956 row_size:0 compaction_score:268401500 compaction_level:-1 partial_update_by_column:false]
71996 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
71997 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
71998 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
71999 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72000 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72001 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72002 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72003 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72004 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72005 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
72006 [seg:0 row:0 del:0 bytes:0 row_size:0 compaction_score:268435456 compaction_level:-1 partial_update_by_column:false]
W20250624 11:34:39.688724 139551225935424 agent_task.cpp:390] clone failed. signature: 310943782
E20250624 11:34:25.492104 139715779958336 threadpool.cpp:468] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
@ 0x6326535 starrocks::get_stack_trace[abi:cxx11]()
@ 0x86ff741 starrocks::ThreadPool::do_submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPoolToken*, starrocks::ThreadPool::Priority)
@ 0x86ffc22 starrocks::ThreadPoolToken::submit_func(std::function<void ()>, starrocks::ThreadPool::Priority)
@ 0x8428926 starrocks::run_publish_version_task(starrocks::ThreadPoolToken*, starrocks::TPublishVersionRequest const&, starrocks::TFinishTaskRequest&, std::unordered_set<starrocks::DataDir*, std::hash<starrocks::DataDir*>, std::equal_to<starrocks::DataDir*>, std::allo^D^S
@ 0x8416f98 starrocks::PublishVersionTaskWorkerPool::_worker_thread_callback(void*)
@ 0xe48f0c4 execute_native_thread_routine
@ 0x7f1369498ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f1369529a04 clone
E20250624 11:34:34.368519 139716702025280 threadpool.cpp:468] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
@ 0x6326535 starrocks::get_stack_trace[abi:cxx11]()
@ 0x86ff741 starrocks::ThreadPool::do_submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPoolToken*, starrocks::ThreadPool::Priority)
@ 0x86ffdc5 starrocks::ThreadPool::submit_func(std::function<void ()>, starrocks::ThreadPool::Priority)
@ 0x79d0a38 starrocks::pipeline::ExecStateReporter::submit(std::function<void ()>&&, bool)
@ 0x79c8d03 starrocks::pipeline::GlobalDriverExecutor::report_exec_state(starrocks::pipeline::QueryContext*, starrocks::pipeline::FragmentContext*, starrocks::Status const&, bool, bool)
@ 0x5298b6d starrocks::pipeline::FragmentContext::report_exec_state_if_necessary()
@ 0x79c75a9 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x87032fe starrocks::ThreadPool::dispatch_thread()
@ 0x86fa569 starrocks::Thread::supervise_thread(void*)
@ 0x7f1369498ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f1369529a04 clone
E20250624 11:34:51.758248 139716668454464 threadpool.cpp:468] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
@ 0x6326535 starrocks::get_stack_trace[abi:cxx11]()
@ 0x86ff741 starrocks::ThreadPool::do_submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPoolToken*, starrocks::ThreadPool::Priority)
@ 0x86ffdc5 starrocks::ThreadPool::submit_func(std::function<void ()>, starrocks::ThreadPool::Priority)
@ 0x79d0a38 starrocks::pipeline::ExecStateReporter::submit(std::function<void ()>&&, bool)
@ 0x79c8d03 starrocks::pipeline::GlobalDriverExecutor::report_exec_state(starrocks::pipeline::QueryContext*, starrocks::pipeline::FragmentContext*, starrocks::Status const&, bool, bool)
@ 0x5298b6d starrocks::pipeline::FragmentContext::report_exec_state_if_necessary()
@ 0x79c75a9 starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x87032fe starrocks::ThreadPool::dispatch_thread()
@ 0x86fa569 starrocks::Thread::supervise_thread(void*)
@ 0x7f1369498ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f1369529a04 clone
E20250624 11:34:52.107829 139716685239872 threadpool.cpp:468] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
@ 0x6326535 starrocks::get_stack_trace[abi:cxx11]()
@ 0x86ff741 starrocks::ThreadPool::do_submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPoolToken*, starrocks::ThreadPool::Priority)
@ 0x86ffdc5 starrocks::ThreadPool::submit_func(std::function<void ()>, starrocks::ThreadPool::Priority)
@ 0x79d099c starrocks::pipeline::ExecStateReporter::submit(std::function<void ()>&&, bool)
@ 0x79c8d03 starrocks::pipeline::GlobalDriverExecutor::report_exec_state(starrocks::pipeline::QueryContext*, starrocks::pipeline::FragmentContext*, starrocks::Status const&, bool, bool)
@ 0x529b59e starrocks::pipeline::FragmentContext::count_down_execution_group(unsigned long)
@ 0x4fbe7ba starrocks::pipeline::PipelineDriver::finalize(starrocks::RuntimeState*, starrocks::pipeline::DriverState, long, long)
@ 0x79c7e8b starrocks::pipeline::GlobalDriverExecutor::_worker_thread()
@ 0x87032fe starrocks::ThreadPool::dispatch_thread()
@ 0x86fa569 starrocks::Thread::supervise_thread(void*)
@ 0x7f1369498ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f1369529a04 clone
E20250624 11:34:52.337756 139715779958336 threadpool.cpp:468] Thread pool failed to create thread: Runtime error: Could not create thread: Resource temporarily unavailable
@ 0x6326535 starrocks::get_stack_trace[abi:cxx11]()
@ 0x86ff741 starrocks::ThreadPool::do_submit(std::shared_ptr<starrocks::Runnable>, starrocks::ThreadPoolToken*, starrocks::ThreadPool::Priority)
@ 0x86ffc22 starrocks::ThreadPoolToken::submit_func(std::function<void ()>, starrocks::ThreadPool::Priority)
@ 0x8428926 starrocks::run_publish_version_task(starrocks::ThreadPoolToken*, starrocks::TPublishVersionRequest const&, starrocks::TFinishTaskRequest&, std::unordered_set<starrocks::DataDir*, std::hash<starrocks::DataDir*>, std::equal_to<starrocks::DataDir*>, std::allo<CB>^S
@ 0x8416f98 starrocks::PublishVersionTaskWorkerPool::_worker_thread_callback(void*)
@ 0xe48f0c4 execute_native_thread_routine
@ 0x7f1369498ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f1369529a04 clone
W20250624 11:39:01.854082 139787398610112 timezone_utils.cpp:93] not found timezone:Africa/Casablanca
W20250624 11:39:01.858060 139787398610112 timezone_utils.cpp:93] not found timezone:Africa/El_Aaiun
W20250624 11:39:21.918707 139786938975808 data_dir.cpp:383] could not find tablet id: 352103156 for rowset: 020000001a7c33f1764ea9eaba931dc98ad7f0669add3d9f, skip loading this rowset
W20250624 11:39:28.682604 139780453049920 heartbeat_server.cpp:205] saas-node12 not equal to to backend localhost 172.17.9.108
E20250624 11:41:10.859719 139782266304064 scan_operator.cpp:465] scan fragment 11b20918-50ad-11f0-ade0-02427eb9345a driver 1 Scan tasks error: Cancelled: Cancelled
be/src/storage/rowset/segment_iterator.cpp:1061 _init()
be/src/storage/tablet_reader.cpp:326 _collect_iter->get_next(chunk)
be/src/exec/pipeline/scan/olap_chunk_source.cpp:518 _prj_iter->get_next(chunk)
E20250624 11:41:10.875904 139782257911360 scan_operator.cpp:465] scan fragment 11b20918-50ad-11f0-ade0-02427eb9345a driver 0 Scan tasks error: Cancelled: Cancelled
be/src/storage/rowset/segment_iterator.cpp:1061 _init()
be/src/storage/tablet_reader.cpp:326 _collect_iter->get_next(chunk)
be/src/exec/pipeline/scan/olap_chunk_source.cpp:518 _prj_iter->get_next(chunk)
W20250624 11:42:33.255474 139781129176640 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=02000000000058826a4d1775922ea04b52b26cace0a2b7a5 tablet: 352113098
W20250624 11:42:33.617697 139781129176640 storage_engine.cpp:1144] traverse_rowset_meta and remove 3520/11693 invalid rowset metas, path:/dbdata1 duration:448ms
W20250624 11:45:37.692829 139781129176640 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=0200000000006a036a4d1775922ea04b52b26cace0a2b7a5 tablet: 352113777
W20250624 11:45:38.255395 139781129176640 storage_engine.cpp:1144] traverse_rowset_meta and remove 344/13284 invalid rowset metas, path:/dbdata1 duration:563ms
W20250624 11:48:42.000827 139781129176640 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=0200000000011c556a4d1775922ea04b52b26cace0a2b7a5 tablet: 352140011
W20250624 11:48:42.933223 139781129176640 storage_engine.cpp:1144] traverse_rowset_meta and remove 1035/14887 invalid rowset metas, path:/dbdata1 duration:932ms
W20250624 11:48:54.735700 139781137569344 rowset.cpp:331] Fail to delete /dbdata1/data/20/352110637/1369509696/020000000000090c6a4d1775922ea04b52b26cace0a2b7a5_0.dat: Not found: /dbdata1/data/20/352110637/1369509696/020000000000090c6a4d1775922ea04b52b26cace0a2b7a5_0.dat: No such file or directory
W20250624 11:48:54.735739 139781137569344 rowset.cpp:331] Fail to delete /dbdata1/data/20/352110637/1369509696/020000000000090c6a4d1775922ea04b52b26cace0a2b7a5_1.dat: Not found: /dbdata1/data/20/352110637/1369509696/020000000000090c6a4d1775922ea04b52b26cace0a2b7a5_1.dat: No such file or directory
W20250624 11:48:54.735747 139781137569344 rowset.cpp:331] Fail to delete /dbdata1/data/20/352110637/1369509696/020000000000090c6a4d1775922ea04b52b26cace0a2b7a5_2.dat: Not found: /dbdata1/data/20/352110637/1369509696/020000000000090c6a4d1775922ea04b52b26cace0a2b7a5_2.dat: No such file or directory
W20250624 11:48:54.735753 139781137569344 rowset.cpp:331] Fail to delete /dbdata1/data/20/352110637/1369509696/020000000000090c6a4d1775922ea04b52b26cace0a2b7a5_3.dat: Not found: /dbdata1/data/20/352110637/1369509696/020000000000090c6a4d1775922ea04b52b26cace0a2b7a5_3.dat: No such file or directory
W20250624 11:48:54.735771 139781137569344 rowset.cpp:331] Fail to delete /dbdata1/data/20/352110637/1369509696/020000000000090c6a4d1775922ea04b52b26cace0a2b7a5_4.dat: Not found: /dbdata1/data/20/352110637/1369509696/020000000000090c6a4d1775922ea04b52b26cace0a2b7a5_4.dat: No such file or directory
W20250624 11:51:48.272094 139781129176640 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=02000000000143cc6a4d1775922ea04b52b26cace0a2b7a5 tablet: 352141185
W20250624 11:51:49.256378 139781129176640 storage_engine.cpp:1144] traverse_rowset_meta and remove 1556/11579 invalid rowset metas, path:/dbdata1 duration:984ms
E20250624 11:52:10.191569 139782299874880 scan_operator.cpp:465] scan fragment 9acf3e19-50ae-11f0-9020-024275dda843 driver 1 Scan tasks error: Cancelled: canceled state
W20250624 11:54:53.903825 139781129176640 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=02000000000237a56a4d1775922ea04b52b26cace0a2b7a5 tablet: 352167376
W20250624 11:54:54.940687 139781129176640 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=0200000000024e506a4d1775922ea04b52b26cace0a2b7a5 tablet: 352177777
W20250624 11:54:55.135315 139781137569344 rowset.cpp:331] Fail to delete /dbdata1/data/330/352130490/1369509696/020000000000bb0d6a4d1775922ea04b52b26cace0a2b7a5_0.dat: Not found: /dbdata1/data/330/352130490/1369509696/020000000000bb0d6a4d1775922ea04b52b26cace0a2b7a5_0.dat: No such file or directory
W20250624 11:54:55.135361 139781137569344 rowset.cpp:331] Fail to delete /dbdata1/data/330/352130490/1369509696/020000000000bb0d6a4d1775922ea04b52b26cace0a2b7a5_1.dat: Not found: /dbdata1/data/330/352130490/1369509696/020000000000bb0d6a4d1775922ea04b52b26cace0a2b7a5_1.dat: No such file or directory
W20250624 11:54:55.135371 139781137569344 rowset.cpp:331] Fail to delete /dbdata1/data/330/352130490/1369509696/020000000000bb0d6a4d1775922ea04b52b26cace0a2b7a5_2.dat: Not found: /dbdata1/data/330/352130490/1369509696/020000000000bb0d6a4d1775922ea04b52b26cace0a2b7a5_2.dat: No such file or directory
W20250624 11:54:55.135381 139781137569344 rowset.cpp:331] Fail to delete /dbdata1/data/330/352130490/1369509696/020000000000bb0d6a4d1775922ea04b52b26cace0a2b7a5_3.dat: Not found: /dbdata1/data/330/352130490/1369509696/020000000000bb0d6a4d1775922ea04b52b26cace0a2b7a5_3.dat: No such file or directory
W20250624 11:54:55.135403 139781137569344 rowset.cpp:331] Fail to delete /dbdata1/data/330/352130490/1369509696/020000000000bb0d6a4d1775922ea04b52b26cace0a2b7a5_4.dat: Not found: /dbdata1/data/330/352130490/1369509696/020000000000bb0d6a4d1775922ea04b52b26cace0a2b7a5_4.dat: No such file or directory
W20250624 11:54:55.700712 139781129176640 storage_engine.cpp:1144] traverse_rowset_meta and remove 74/14855 invalid rowset metas, path:/dbdata1 duration:1798ms
W20250624 11:58:03.351288 139781129176640 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=0200000000023e466a4d1775922ea04b52b26cace0a2b7a5 tablet: 352169982
W20250624 11:58:04.543383 139781129176640 storage_engine.cpp:1124] tablet not found, remove rowset meta, rowset_id=020000001a7bc841764ea9eaba931dc98ad7f0669add3d9f tablet: 352074860
W20250624 11:58:04.779924 139781129176640 storage_engine.cpp:1144] traverse_rowset_meta and remove 2798/16273 invalid rowset metas, path:/dbdata1 duration:1429ms
E20250624 11:58:31.692775 139782299874880 scan_operator.cpp:465] scan fragment 7e313c10-50af-11f0-9020-024275dda844 driver 1 Scan tasks error: Cancelled: canceled state
【背景】做过哪些操作？
【业务影响】
【是否存算分离】
【StarRocks版本】例如：3.3.12
【集群规模】例如：3fe（3 follower+0observer）+3be（fe与be混部）
【机器信息】CPU虚拟核/内存/网卡，例如：48C/64G/万兆
【联系方式】为了在解决问题过程中能及时联系到您获取一些日志信息，请补充下您的联系方式，例如：社区群16-可乐鸡或者邮箱，谢谢
【附件】
- fe.log/beINFO/相应截图
- 慢查询：
- Profile信息
- 并行度：show variables like ‘%parallel_fragment_exec_instance_num%’;
- pipeline是否开启：show variables like ‘%pipeline%’;
- be节点cpu和内存使用率截图
- 查询报错：
- query_dump，怎么获取query_dump文件
- be crash
- be.out
- 外表查询报错
- be.out和fe.warn.log
