【详述】be节点重启失败
【背景】运行一段时间后出现的问题；现在只有1个节点出现问题，其他节点正常
【业务影响】
【StarRocks版本】例如：2.2.10
【集群规模】3FE + 21BE
【机器信息】阿里云ECS 64C/128G/万兆
【联系方式】社区群3-如斯
【附件】
be.info
07:01:24.627990 137373 daemon.cpp:267] version 2.2.10 RELEASE (build b101696)
Built on 2022-11-25 17:56:58 by StarRocks@docker
I0202 07:01:24.636965 137373 mem_info.cpp:78] Physical Memory: 247.78 GB
I0202 07:01:24.636974 137373 daemon.cpp:273] Cpu Info:
Model: AMD EPYC 7T83 64-Core Processor
Cores: 64
Max Possible Cores: 64
L1 Cache: 32.00 KB (Line: 64.00 B)
L2 Cache: 512.00 KB (Line: 64.00 B)
L3 Cache: 256.00 MB (Line: 64.00 B)
Hardware Supports:
ssse3
sse4_1
sse4_2
popcnt
avx
avx2
Numa Nodes: 2
Numa Nodes of Cores: 0->0 | 1->0 | 2->0 | 3->0 | 4->0 | 5->0 | 6->0 | 7->0 | 8->0 | 9->0 | 10->0 | 11->0 | 12->0 | 13->0 | 14->0 | 15->0 | 16->0 | 17->0 | 18->0 | 19->0 | 20->0 | 21->0 | 22->0 | 23->0 | 24->0 | 25->0 | 26->0 | 27->0 | 28->0 | 29->0 | 30->0 | 31->0 | 32->1 | 33->1 | 34->1 | 35->1 | 36->1 | 37->1 | 38->1 | 39->1 | 40->1 | 41->1 | 42->1 | 43->1 | 44->1 | 45->1 | 46->1 | 47->1 | 48->1 | 49->1 | 50->1 | 51->1 | 52->1 | 53->1 | 54->1 | 55->1 | 56->1 | 57->1 | 58->1 | 59->1 | 60->1 | 61->1 | 62->1 | 63->1 |
I0202 07:01:24.637005 137373 daemon.cpp:274] Disk Info:
Num disks 6: vda, vdb, vdc, vdd, vde, vdf
I0202 07:01:24.637012 137373 daemon.cpp:275] Mem Info: 247.78 GB
I0202 07:01:24.676093 137373 daemon.cpp:254] Minidump is disabled
I0202 07:01:24.676105 137373 backend_options.cpp:92] priority cidrs in conf: 10.65.0.0/16
I0202 07:01:24.676214 137373 backend_options.cpp:80] local host ip=10.65.23.175
I0202 07:01:24.676611 137390 daemon.cpp:182] Current memory statistics: process(3701456), query_pool(0), load(0), tablet_meta(0), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), clone(0), consistency(0)
I0202 07:01:24.676990 137373 storage_engine.cpp:82] Opening storage engine using uid=40450d44c64b18e6-d7902dec87ad2291
I0202 07:01:24.677917 137392 data_dir.cpp:726] path: /data_b1 total capacity: 1082119344128, available capacity: 743390126080
I0202 07:01:24.677985 137393 data_dir.cpp:726] path: /data_c1 total capacity: 1082119344128, available capacity: 731278188544
I0202 07:01:24.677999 137394 data_dir.cpp:726] path: /data_d1 total capacity: 1082119344128, available capacity: 731358146560
I0202 07:01:24.678020 137395 data_dir.cpp:726] path: /data_e1 total capacity: 1082119344128, available capacity: 728353898496
I0202 07:01:24.678098 137396 data_dir.cpp:726] path: /data_f1 total capacity: 1082119344128, available capacity: 739329265664
I0202 07:01:24.679280 137394 data_dir.cpp:261] path: /data_d1, hash: 5001438776404103191
I0202 07:01:24.679877 137393 data_dir.cpp:261] path: /data_c1, hash: -2484366089668085966
I0202 07:01:24.680056 137396 data_dir.cpp:261] path: /data_f1, hash: -584851486764415702
I0202 07:01:24.680534 137392 data_dir.cpp:261] path: /data_b1, hash: -6882387974670824587
I0202 07:01:24.680723 137395 data_dir.cpp:261] path: /data_e1, hash: -8751533704141381708
W0202 07:01:24.709842 137394 kv_store.cpp:102] Fail to list column families, reason:Corruption: checksum mismatch
W0202 07:01:24.712327 137394 data_dir.cpp:270] Fail to init meta store: Corruption: Corruption: checksum mismatch
W0202 07:01:24.712337 137394 data_dir.cpp:101] _init_meta failed, error: Corruption: Corruption: checksum mismatch
W0202 07:01:24.712344 137394 storage_engine.cpp:219] Store load failed, status=Corruption: Corruption: checksum mismatch, path=/data_d1
W0202 07:01:25.762068 137373 storage_engine.cpp:158] _init_store_map failed, error: Internal error: init path failed, error=Corruption: Corruption: checksum mismatch;
W0202 07:01:25.762082 137373 storage_engine.cpp:84] open engine failed, error: Internal error: init path failed, error=Corruption: Corruption: checksum mismatch;
F0202 07:01:25.762159 137373 starrocks_main.cpp:209] fail to open StorageEngine, res=init path failed, error=Corruption: Corruption: checksum mismatch;
be.out
query_id:00000000-0000-0000-0000-000000000000
*** Aborted at 1675292428 (unix time) try “date -d @1675292428” if you are using GNU date ***
PC: @ 0x385add1 Curl_clone_primary_ssl_config
*** SIGSEGV (@0x400000000008) received by PID 163102 (TID 0x7fdeb220a700) from PID 8; stack trace: ***
@ 0x3d0e5d2 google::(anonymous namespace)::FailureSignalHandler()
@ 0x7fdfaea73630 (unknown)
@ 0x385add1 Curl_clone_primary_ssl_config
@ 0x385500e Curl_connect
@ 0x383e431 multi_runsingle
@ 0x383f5f3 curl_multi_perform
@ 0x383034b curl_easy_perform
@ 0x22d6f03 starrocks::HttpClient::execute()
@ 0x22d7287 starrocks::HttpClient::download()
@ 0x31bb5c5 ZNSt17_Function_handlerIFN9starrocks6StatusEPNS0_10HttpClientEEZNS0_15EngineCloneTask15_download_filesEPNS0_7DataDirERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESF_EUlS3_E1_E9_M_invokeERKSt9_Any_dataOS3
@ 0x22d7611 starrocks::HttpClient::execute_with_retry()
@ 0x31bf777 starrocks::EngineCloneTask::_download_files()
@ 0x31c04ad starrocks::EngineCloneTask::_clone_copy()
@ 0x31c3894 starrocks::EngineCloneTask::_do_clone()
@ 0x31c49d7 starrocks::EngineCloneTask::execute()
@ 0x1987bde starrocks::StorageEngine::execute_task()
@ 0x2311a0e starrocks::TaskWorkerPool::_clone_worker_thread_callback()
@ 0x5746810 execute_native_thread_routine
@ 0x7fdfaea6bea5 start_thread
@ 0x7fdfae086b0d __clone
@ 0x0 (unknown)
start time: Thu Feb 2 07:01:24 CST 2023
start time: Thu Feb 2 07:01:24 CST 2023
*** Check failure stack trace: ***
@ 0x3d01fad google::LogMessage::Fail()
@ 0x3d0441f google::LogMessage::SendToLog()
@ 0x3d01afe google::LogMessage::Flush()
@ 0x3d04a29 google::LogMessageFatal::~LogMessageFatal()
@ 0x17fc508 main
@ 0x7f55828f8555 __libc_start_main
@ 0x18e971e (unknown)
@ (nil) (unknown)
dmesg -T 没有磁盘报错， d1盘读写正常