集群cn节点一直重启,且无日志文件

【详述】在k8s集群里使用starrocks-operator安装了3fe+1cn的集群

版本是:starrocks/fe-ubuntu:3.2.4
存算分离模式,日志用efs挂载存储。cn存储使用的是aws s3
cn配置文件属性是
sys_log_level = INFO

thrift_port = 9060
be_http_port = 8040
heartbeat_service_port = 9050
brpc_port = 8060

#存储盘
storage_root_path = ${STARROCKS_HOME}/storage

cn重启前日志报错信息:
E0322 19:53:52.716013 818 block_file.cpp:184] fail to open block file, reason: Disk quota exceeded
I0322 19:53:56.591416 171 daemon.cpp:191] Current memory statistics: process(450855808), query_pool(0), load(0), metadata(0), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), clone(0), consistency(0), datacache(0)
I0322 19:53:56.753221 933 load_channel_mgr.cpp:249] Memory consumption(bytes) limit=32266706337 current=0 peak=0
I0322 19:53:58.701738 986 heartbeat_server.cpp:77] get heartbeat from FE.host:starrocks-prod-fe-0.starrocks-prod-fe-search.starrocks.svc.cluster.local, port:9020, cluster id:1414201577, run_mode:SHARED_DATA, counter:13
I0322 19:53:58.760221 862 tablet_manager.cpp:971] Report all 0 tablets info. max_tablet_rowset_num:0
I0322 19:54:11.617100 171 daemon.cpp:191] Current memory statistics: process(446661456), query_pool(0), load(0), metadata(0), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), clone(0), consistency(0), datacache(0)
I0322 19:54:26.623116 171 daemon.cpp:191] Current memory statistics: process(442467024), query_pool(0), load(0), metadata(0), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), clone(0), consistency(0), datacache(0)
E0322 19:54:32.970986 818 cache_manager.cc:171] INTERNAL: fail to init star cache, fail to open block file, reason: Disk quota exceeded
I0322 19:54:32.976447 818 lru_container.cpp:401] Successfully prune container, clean 0 entries.
I0322 19:54:32.976495 818 lru_container.cpp:401] Successfully prune container, clean 0 entries.
F0322 19:54:33.702685 986 doubly_buffered_data.h:248] Check failed: false Invalid id=-1
*** Check failure stack trace: ***
3.2.4 RELEASE (build 613f0b5)
query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000
tracker:process consumption: 104781496
tracker:query_pool consumption: 0
tracker:network_receive_bytes consumption: 0
tracker:load consumption: 0
tracker:metadata consumption: 0
tracker:tablet_metadata consumption: 0
tracker:rowset_metadata consumption: 0
tracker:segment_metadata consumption: 0
tracker:column_metadata consumption: 0
tracker:tablet_schema consumption: 0
tracker:segment_zonemap consumption: 0
tracker:short_key_index consumption: 0
tracker:column_zonemap_index consumption: 0
tracker:ordinal_index consumption: 0
tracker:bitmap_index consumption: 0
tracker:bloom_filter_index consumption: 0
tracker:compaction consumption: 0
tracker:schema_change consumption: 0
tracker:column_pool consumption: 0
tracker:page_cache consumption: 0
tracker:update consumption: 0
tracker:chunk_allocator consumption: 0
tracker:clone consumption: 0
tracker:consistency consumption: 0
tracker:datacache consumption: 0
tracker:replication consumption: 0
*** Aborted at 1711108473 (unix time) try “date -d @1711108473” if you are using GNU date ***
PC: @ 0x7fefdab629fc pthread_kill
*** SIGABRT (@0x1b) received by PID 27 (TID 0x7fedd422b640) from PID 27; stack trace: ***
@ 0x7f87a9a google::(anonymous namespace)::FailureSignalHandler()
@ 0x7fefdab0e520 (unknown)
@ 0x7fefdab629fc pthread_kill
@ 0x7fefdab0e476 raise
@ 0x7fefdaaf47f3 abort
@ 0x3d6e8f2 starrocks::failure_function()
@ 0x7f76031 google::LogMessage::Fail()
@ 0x7f7868f google::LogMessage::SendToLog()
@ 0x7f75b70 google::LogMessage::Flush()
@ 0x7f78cfd google::LogMessageFatal::~LogMessageFatal()
@ 0x3eb87ee butil::DoublyBufferedData<>::Read()
@ 0x3eb7992 starrocks::get_master_info()
@ 0x6a2d7f2 starrocks::HeartbeatServer::compare_master_info()
@ 0x6a2e9a1 starrocks::HeartbeatServer::heartbeat()
@ 0x6cdfa09 starrocks::HeartbeatServiceProcessor::process_heartbeat()
@ 0x6ce03f0 starrocks::HeartbeatServiceProcessor::dispatchCall()
F0322 19:54:33.771178 863 doubly_buffered_data.h:248] Check failed: false Invalid id=-1
*** Check failure stack trace: ***
@ 0x6c8431b apache::thrift::TDispatchProcessor::process()
@ 0x7f64d79 apache::thrift::server::TConnectedClient::run()
@ 0x7f65c58 apache::thrift::server::TThreadedServer::TConnectedClientRunner::run()
@ 0x7f680ef apache::thrift::concurrency::thread::threadMain()
@ 0x7f426b9 std::thread::_State_impl<>::_M_run()
@ 0xa5fae94 execute_native_thread_routine
@ 0x7fefdab60ac3 (unknown)
@ 0x7fefdabf2850 (unknown)
F0322 19:54:33.771178 863 doubly_buffered_data.h:248] Check failed: false Invalid id=-1F0322 19:54:33.821220 864 doubly_buffered_data.h:248] Check failed: false Invalid id=-1
*** Check failure stack trace: ***
@ 0x0 (unknown)
/opt/starrocks/cn_entrypoint.sh: line 185: 27 Aborted (core dumped) $STARROCKS_HOME/bin/start_cn.sh $addition_args
cn节点终端无日志文件,控制台一直在输出
I0322 18:42:03.999104 818 block_file.cpp:39] open block file success, fd: 5770, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5540
I0322 18:42:04.001160 818 block_file.cpp:39] open block file success, fd: 5771, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5541
I0322 18:42:04.003055 818 block_file.cpp:39] open block file success, fd: 5772, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5542
I0322 18:42:04.004982 818 block_file.cpp:39] open block file success, fd: 5773, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5543
I0322 18:42:04.007197 818 block_file.cpp:39] open block file success, fd: 5774, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5544
I0322 18:42:04.009161 818 block_file.cpp:39] open block file success, fd: 5775, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5545
I0322 18:42:04.011238 818 block_file.cpp:39] open block file success, fd: 5776, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5546
I0322 18:42:04.013244 818 block_file.cpp:39] open block file success, fd: 5777, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5547
I0322 18:42:04.015237 818 block_file.cpp:39] open block file success, fd: 5778, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5548
I0322 18:42:04.017570 818 block_file.cpp:39] open block file success, fd: 5779, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5549
I0322 18:42:04.019467 818 block_file.cpp:39] open block file success, fd: 5780, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5550
I0322 18:42:04.022570 818 block_file.cpp:39] open block file success, fd: 5781, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5551
I0322 18:42:04.028532 818 block_file.cpp:39] open block file success, fd: 5782, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5552
I0322 18:42:04.031273 818 block_file.cpp:39] open block file success, fd: 5784, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5553
I0322 18:42:04.033263 818 block_file.cpp:39] open block file success, fd: 5783, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5554
I0322 18:42:04.035231 818 block_file.cpp:39] open block file success, fd: 5785, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5555
I0322 18:42:04.037367 818 block_file.cpp:39] open block file success, fd: 5786, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5556
I0322 18:42:04.039461 818 block_file.cpp:39] open block file success, fd: 5787, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5557
I0322 18:42:04.041402 818 block_file.cpp:39] open block file success, fd: 5788, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5558
I0322 18:42:04.043502 818 block_file.cpp:39] open block file success, fd: 5789, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5559
I0322 18:42:04.050068 818 block_file.cpp:39] open block file success, fd: 5790, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5560
I0322 18:42:04.052294 818 block_file.cpp:39] open block file success, fd: 5791, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5561
I0322 18:42:04.054244 818 block_file.cpp:39] open block file success, fd: 5792, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5562
I0322 18:42:04.059235 818 block_file.cpp:39] open block file success, fd: 5793, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5563
I0322 18:42:04.061553 818 block_file.cpp:39] open block file success, fd: 5794, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5564
I0322 18:42:04.063571 818 block_file.cpp:39] open block file success, fd: 5795, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5565
I0322 18:42:04.065615 818 block_file.cpp:39] open block file success, fd: 5796, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5566
I0322 18:42:04.068243 818 block_file.cpp:39] open block file success, fd: 5797, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5567
I0322 18:42:04.074366 818 block_file.cpp:39] open block file success, fd: 5798, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5568
I0322 18:42:04.076813 818 block_file.cpp:39] open block file success, fd: 5799, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5569
I0322 18:42:04.078981 818 block_file.cpp:39] open block file success, fd: 5800, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5570
I0322 18:42:04.080926 818 block_file.cpp:39] open block file success, fd: 5801, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5571
I0322 18:42:04.083333 818 block_file.cpp:39] open block file success, fd: 5802, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5572
I0322 18:42:04.085647 818 block_file.cpp:39] open block file success, fd: 5803, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5573
I0322 18:42:04.090508 818 block_file.cpp:39] open block file success, fd: 5804, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5574
I0322 18:42:04.092800 818 block_file.cpp:39] open block file success, fd: 5805, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5575
I0322 18:42:04.094936 818 block_file.cpp:39] open block file success, fd: 5806, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5576
I0322 18:42:04.097187 818 block_file.cpp:39] open block file success, fd: 5807, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5577
I0322 18:42:04.100033 818 block_file.cpp:39] open block file success, fd: 5808, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5578
I0322 18:42:04.102186 818 block_file.cpp:39] open block file success, fd: 5809, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5579
I0322 18:42:04.104209 818 block_file.cpp:39] open block file success, fd: 5810, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5580
I0322 18:42:04.106448 818 block_file.cpp:39] open block file success, fd: 5811, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5581
I0322 18:42:04.108878 818 block_file.cpp:39] open block file success, fd: 5812, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5582
I0322 18:42:04.111178 818 block_file.cpp:39] open block file success, fd: 5813, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5583
I0322 18:42:04.113291 818 block_file.cpp:39] open block file success, fd: 5814, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5584
I0322 18:42:04.115427 818 block_file.cpp:39] open block file success, fd: 5815, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5585
I0322 18:42:04.117396 818 block_file.cpp:39] open block file success, fd: 5816, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5586
I0322 18:42:04.119941 818 block_file.cpp:39] open block file success, fd: 5817, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5587
I0322 18:42:04.123802 818 block_file.cpp:39] open block file success, fd: 5818, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5588
I0322 18:42:04.128101 818 block_file.cpp:39] open block file success, fd: 5819, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5589
I0322 18:42:04.130535 818 block_file.cpp:39] open block file success, fd: 5820, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5590
I0322 18:42:04.132577 818 block_file.cpp:39] open block file success, fd: 5821, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5591
I0322 18:42:04.134697 818 block_file.cpp:39] open block file success, fd: 5822, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5592
I0322 18:42:04.136932 818 block_file.cpp:39] open block file success, fd: 5823, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5593
I0322 18:42:04.141211 818 block_file.cpp:39] open block file success, fd: 5824, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5594
I0322 18:42:04.148277 818 block_file.cpp:39] open block file success, fd: 5825, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5595
I0322 18:42:04.150408 818 block_file.cpp:39] open block file success, fd: 5826, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5596
I0322 18:42:04.152289 818 block_file.cpp:39] open block file success, fd: 5827, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5597
I0322 18:42:04.154373 818 block_file.cpp:39] open block file success, fd: 5828, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5598
I0322 18:42:04.156525 818 block_file.cpp:39] open block file success, fd: 5829, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5599
I0322 18:42:04.159109 818 block_file.cpp:39] open block file success, fd: 5830, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5600
I0322 18:42:04.161679 818 block_file.cpp:39] open block file success, fd: 5831, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5601
I0322 18:42:04.169706 818 block_file.cpp:39] open block file success, fd: 5832, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5602
I0322 18:42:04.172482 818 block_file.cpp:39] open block file success, fd: 5833, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5603
I0322 18:42:04.174669 818 block_file.cpp:39] open block file success, fd: 5834, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5604
I0322 18:42:04.176954 818 block_file.cpp:39] open block file success, fd: 5835, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5605
I0322 18:42:04.178861 818 block_file.cpp:39] open block file success, fd: 5836, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5606
I0322 18:42:04.181288 818 block_file.cpp:39] open block file success, fd: 5837, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5607
I0322 18:42:04.183373 818 block_file.cpp:39] open block file success, fd: 5838, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5608
I0322 18:42:04.185381 818 block_file.cpp:39] open block file success, fd: 5839, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5609
I0322 18:42:04.187572 818 block_file.cpp:39] open block file success, fd: 5840, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5610
I0322 18:42:04.190003 818 block_file.cpp:39] open block file success, fd: 5841, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5611
I0322 18:42:04.192080 818 block_file.cpp:39] open block file success, fd: 5842, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5612
I0322 18:42:04.194586 818 block_file.cpp:39] open block file success, fd: 5843, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5613
I0322 18:42:04.197239 818 block_file.cpp:39] open block file success, fd: 5844, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5614
I0322 18:42:04.199297 818 block_file.cpp:39] open block file success, fd: 5845, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5615
I0322 18:42:04.203110 818 block_file.cpp:39] open block file success, fd: 5846, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5616
I0322 18:42:04.205421 818 block_file.cpp:39] open block file success, fd: 5847, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5617
I0322 18:42:04.207305 818 block_file.cpp:39] open block file success, fd: 5848, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5618
I0322 18:42:04.209707 818 block_file.cpp:39] open block file success, fd: 5849, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5619
I0322 18:42:04.211706 818 block_file.cpp:39] open block file success, fd: 5850, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5620
I0322 18:42:04.213850 818 block_file.cpp:39] open block file success, fd: 5851, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5621
I0322 18:42:04.215863 818 block_file.cpp:39] open block file success, fd: 5852, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5622
I0322 18:42:04.217952 818 block_file.cpp:39] open block file success, fd: 5853, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5623
I0322 18:42:04.219873 818 block_file.cpp:39] open block file success, fd: 5854, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5624
I0322 18:42:04.222472 818 block_file.cpp:39] open block file success, fd: 5855, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5625
I0322 18:42:04.224503 818 block_file.cpp:39] open block file success, fd: 5856, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5626
I0322 18:42:04.226833 818 block_file.cpp:39] open block file success, fd: 5857, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5627
I0322 18:42:04.228812 818 block_file.cpp:39] open block file success, fd: 5858, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5628
I0322 18:42:04.230868 818 block_file.cpp:39] open block file success, fd: 5859, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5629
I0322 18:42:04.237690 818 block_file.cpp:39] open block file success, fd: 5860, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5630
I0322 18:42:04.240120 818 block_file.cpp:39] open block file success, fd: 5861, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5631
I0322 18:42:04.242077 818 block_file.cpp:39] open block file success, fd: 5862, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5632
I0322 18:42:04.245280 818 block_file.cpp:39] open block file success, fd: 5863, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5633
I0322 18:42:04.247232 818 block_file.cpp:39] open block file success, fd: 5864, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5634
I0322 18:42:04.249378 818 block_file.cpp:39] open block file success, fd: 5865, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5635
I0322 18:42:04.251410 818 block_file.cpp:39] open block file success, fd: 5866, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5636
I0322 18:42:04.253487 818 block_file.cpp:39] open block file success, fd: 5867, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5637
I0322 18:42:04.255381 818 block_file.cpp:39] open block file success, fd: 5868, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5638
I0322 18:42:04.257299 818 block_file.cpp:39] open block file success, fd: 5869, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5639
I0322 18:42:04.259256 818 block_file.cpp:39] open block file success, fd: 5870, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5640
I0322 18:42:04.261289 818 block_file.cpp:39] open block file success, fd: 5871, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5641
I0322 18:42:04.263274 818 block_file.cpp:39] open block file success, fd: 5872, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5642
I0322 18:42:04.265256 818 block_file.cpp:39] open block file success, fd: 5873, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5643
I0322 18:42:04.267156 818 block_file.cpp:39] open block file success, fd: 5874, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5644
I0322 18:42:04.269207 818 block_file.cpp:39] open block file success, fd: 5875, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5645
I0322 18:42:04.271163 818 block_file.cpp:39] open block file success, fd: 5876, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5646
I0322 18:42:04.273115 818 block_file.cpp:39] open block file success, fd: 5877, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5647
I0322 18:42:04.275066 818 block_file.cpp:39] open block file success, fd: 5878, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5648
I0322 18:42:04.277369 818 block_file.cpp:39] open block file success, fd: 5879, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5649
I0322 18:42:04.279289 818 block_file.cpp:39] open block file success, fd: 5880, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5650
I0322 18:42:04.281165 818 block_file.cpp:39] open block file success, fd: 5881, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5651
I0322 18:42:04.283036 818 block_file.cpp:39] open block file success, fd: 5882, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5652
I0322 18:42:04.285176 818 block_file.cpp:39] open block file success, fd: 5883, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5653
I0322 18:42:04.287241 818 block_file.cpp:39] open block file success, fd: 5884, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5654
I0322 18:42:04.289196 818 block_file.cpp:39] open block file success, fd: 5885, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5655
I0322 18:42:04.291220 818 block_file.cpp:39] open block file success, fd: 5886, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5656
I0322 18:42:04.293545 818 block_file.cpp:39] open block file success, fd: 5887, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5657
I0322 18:42:04.295540 818 block_file.cpp:39] open block file success, fd: 5888, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5658
I0322 18:42:04.297533 818 block_file.cpp:39] open block file success, fd: 5889, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5659
I0322 18:42:04.299674 818 block_file.cpp:39] open block file success, fd: 5890, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5660
I0322 18:42:04.301604 818 block_file.cpp:39] open block file success, fd: 5891, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5661
I0322 18:42:04.303478 818 block_file.cpp:39] open block file success, fd: 5892, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5662
I0322 18:42:04.305575 818 block_file.cpp:39] open block file success, fd: 5893, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5663
I0322 18:42:04.307874 818 block_file.cpp:39] open block file success, fd: 5894, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5664
I0322 18:42:04.310344 818 block_file.cpp:39] open block file success, fd: 5895, path: /opt/starrocks/cn/storage/starlet_cache/star_cache/blockfile_5665

starlet_use_star_cache 在3.2.3版本后默认开启. 上面日志显示磁盘空间不足, 导致CN进程退出, 在退出过程中, 触发coredump.

这个问题我解决了,是因为CN节点存储外挂了efs,然后就启动后一直读取efs上的文件,这里一直在打开,我不是很理解。但是最后我去掉了cn节点的外挂efs,cn服务启动成功,日志也成功打印。如果要用efs的外挂磁盘,那我应该在cn节点配置文件中怎么设置??

storageVolumes:
  - name: cn-storage-data
    storageClassName: "starrocks-efs-sc"  # you can remove this line if you want to use the default storage class
    storageSize: 100Gi   # the size of storage volume for data
    mountPath: /opt/starrocks/cn/storage  # the path of data
  - name: cn-storage-log
    storageClassName: "starrocks-efs-sc"  # you can remove this line if you want to use the default storage class
    storageSize: 10Gi  # the size of storage volume for log
    mountPath: /opt/starrocks/cn/log  # the path of log

问题就是出在这个cn-storage-data的外挂efs

做好磁盘空间规划, 默认StarCache可以用80%的磁盘空间, 可以适当调整这个默认参数: https://docs.starrocks.io/docs/administration/management/BE_configuration/#starlet_star_cache_disk_size_percent

问题是我最后去掉了这个外挂磁盘,采用本地ec2的磁盘为什么这个问题就消失了呢?

那要看看容量限制是否起作用, 一般来说挂ec2盘是以emptyDir形式, 重启数据就没了, 也就不存在启动后容量超了的错误.

感觉我俩好像谈的问题不在一个点上,我先是分配了cn节点使用efs,给了100G的空间,按你的说法是默认StarCache可以用80%的磁盘空间,那使用80G呗,为什么还会出现磁盘空间不足这个问题。不就已经很奇怪了嘛,而且挂盘的目的只是为了本地缓存,这个服务只是测试启动就爆空间不足我的确不懂,因为我这个数据库里面一点数据都没有,只是一个启动cn节点就爆内存不足。。。

是不是efs上有残留历史数据写满了? 如果有条件, 可以尝试挂一下对应的pvc进去看看.

内存不足是指哪个错误?