【详述】starrocks版本是3.1.2-4f3a2ee,配置一台fe,三台be,周二晚上使用了date_diff函数导致所有be挂掉了,重启be的时候有两台启动成功,其中有一台be启动不了,查看be.INFO报以下错误:
I1228 16:47:40.336633 8628 daemon.cpp:288] Minidump is disabled
I1228 16:47:40.336795 8628 backend_options.cpp:77] localhost 10.15.22.2
I1228 16:47:40.336867 8628 exec_env.cpp:470] Set storage page cache size 21903109005
I1228 16:47:40.337536 8648 daemon.cpp:211] Current memory statistics: process(0), query_pool(0), load(0), metadata(0), compaction(0), schema_change(0), column_pool(0), page_cache(0), update(0), chunk_allocator(0), clone(0), consistency(0)
I1228 16:47:40.338476 8650 data_dir.cpp:134] path: /data2/soft/StarRocks-3.1.2/be/storage, hash: 1365762131410092910
W1228 16:47:40.344936 8650 kv_store.cpp:116] Fail to open RocksDB, reason:Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data2/soft/StarRocks-3.1.2/be/storage/meta/000058.ldb: No such file or directory, path:/data2/soft/StarRocks-3.1.2/be/storage/meta
W1228 16:47:40.349032 8650 kv_store.cpp:145] Fail to open rocksdb, reason:Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data2/soft/StarRocks-3.1.2/be/storage/meta/000058.ldb: No such file or directory
W1228 16:47:40.349054 8650 data_dir.cpp:143] Fail to init meta store: Corruption: Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data2/soft/StarRocks-3.1.2/be/storage/meta/000058.ldb: No such file or directory
W1228 16:47:40.349062 8650 data_dir.cpp:99] _init_meta failed, error: Corruption: Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data2/soft/StarRocks-3.1.2/be/storage/meta/000058.ldb: No such file or directory
W1228 16:47:40.349073 8650 storage_engine.cpp:266] Store load failed, status=Corruption: Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data2/soft/StarRocks-3.1.2/be/storage/meta/000058.ldb: No such file or directory, path=/data2/soft/StarRocks-3.1.2/be/storage
W1228 16:47:40.349180 8628 storage_engine.cpp:186] _init_store_map failed, error: Internal error: init path failed, error=Corruption: Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data2/soft/StarRocks-3.1.2/be/storage/meta/000058.ldb: No such file or directory;
W1228 16:47:40.349206 8628 storage_engine.cpp:101] open engine failed, error: Internal error: init path failed, error=Corruption: Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data2/soft/StarRocks-3.1.2/be/storage/meta/000058.ldb: No such file or directory;
F1228 16:47:40.349272 8628 starrocks_main.cpp:288] fail to open StorageEngine, res=init path failed, error=Corruption: Corruption: Corruption: IO error: No such file or directory: While open a file for random read: /data2/soft/StarRocks-3.1.2/be/storage/meta/000058.ldb: No such file or directory;
【背景】在使用date_diff函数之后所有be节点都挂掉了,导致starrocks不可用。重启be导致的问题,重启失败的be我并不能保证是因为date_diff挂掉,还是之前已经挂掉了。
【业务影响】
【是否存算分离】否
【StarRocks版本】3.1.2-4f3a2ee
【集群规模】例如:3fe(1 follower+2observer)+3be(fe与be混部)
【机器信息】CPU虚拟核/内存/网卡,16C/128G/-
【联系方式】社区群16-程*梅
【附件】
be.out (15.6 KB)
be.WARNING 发下看看
老师你好,我的问题能重现吗?或者有什么办法能恢复这台be服务呢?