【详述】
【背景】创建GCS存储桶和仓库均成功,通过export 和BACKUP SNAPSHOT功能备份异常
【业务影响】
【StarRocks版本】2.5.6
【集群规模】例如：3fe（1 follower+2observer）+3be
【机器信息】CPU虚拟核/内存/网卡，例如：16C/64G/万兆
【附件】
您任务中使用的用户有在GCS上 delete 的权限么？
有的,是storage admin权限
export 和BACKUP SNAPSHOT的命令方便脱敏发下么
export 命令:
EXPORT TABLE orders
TO “s3a://zrktest/backupTest3/orders1/”
WITH BROKER
(
“fs.s3a.access.key” = “xxx”,
“fs.s3a.secret.key” = “xxx”,
“fs.s3a.endpoint” = “http://storage.googleapis.com”,
“fs.s3a.connection.ssl.enabled” = “false”
)
backup snapshot 命令:
BACKUP SNAPSHOT stage.test_lme_trace_order_trace_info
TO gcs_repo
ON (test_lme_trace_order_trace_info);
aws.s3.access_key
aws.s3.secret_key
aws.s3.endpoint
aws.s3.enable_path_style_access"=“true”,
aws.s3.enable_ssl"=“false” 您用这几个鉴权方式试一下
我的日志， 分享给你看下
2023-08-25T17:48:15.700+0800 info conn starrocks success
2023-08-25T17:48:15.709+0800 info
CREATE REPOSITORY StarRocks
WITH broker hdfs_broker
ON LOCATION "COS桶路径"
PROPERTIES (
"fs.cosn.userinfo.secretId" = "ak",
"fs.cosn.userinfo.secretKey" = "sk",
"fs.cosn.bucket.endpoint_suffix" = "endpoint"
)
2023-08-25T17:48:17.732+0800 info 创建存储快照仓库成功!
2023-08-25T17:48:17.732+0800 info BACKUP SNAPSHOT demo.label_demo_ads_ec_20230825174817 TO StarRocks ON (ads_ec) PROPERTIES ("type" = "full")
2023-08-25T17:48:18.373+0800 info 提交全量备份数据库快照~
2023-08-25T17:48:18.373+0800 info 查询备份快照状态...
2023-08-25T17:48:18.395+0800 info
----------------------------------------------
JobId: -1,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: PENDING,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: ,
UploadFinishedTime: ,
FinishedTime: ,
UnfinishedTasks: ,
Progress: ,
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:20.406+0800 info
----------------------------------------------
JobId: -1,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: PENDING,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: ,
UploadFinishedTime: ,
FinishedTime: ,
UnfinishedTasks: ,
Progress: ,
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:22.417+0800 info
----------------------------------------------
JobId: 77582501,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: SNAPSHOTING,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: ,
UploadFinishedTime: ,
FinishedTime: ,
UnfinishedTasks: ,
Progress: ,
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:24.429+0800 info
----------------------------------------------
JobId: 77582501,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: UPLOAD_SNAPSHOT,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: 2023-08-25 17:48:24,
UploadFinishedTime: ,
FinishedTime: ,
UnfinishedTasks: ,
Progress: ,
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:26.440+0800 info
----------------------------------------------
JobId: 77582501,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: UPLOAD_SNAPSHOT,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: 2023-08-25 17:48:24,
UploadFinishedTime: ,
FinishedTime: ,
UnfinishedTasks: ,
Progress: ,
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:28.451+0800 info
----------------------------------------------
JobId: 77582501,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: UPLOADING,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: 2023-08-25 17:48:24,
UploadFinishedTime: ,
FinishedTime: ,
UnfinishedTasks: 77582541=66938448, 77582537=11395862, 77582536=11395862, 77582535=11395862, 77582534=11395862, 77582533=11395863, 77582532=11395863, 77582531=11395863, 77582530=11395863, 77582529=11395866, 77582525=11395864, 77582521=11395865, 77582517=66938446, 77582513=7643094, 77582509=66938447, 77582508=66938447, 77582506=66938447, 77582505=11395868,
Progress: [77582541: 0/0], [77582536: 0/0], [77582535: 0/0], [77582534: 0/0], [77582532: 0/0], [77582531: 0/0], [77582530: 0/0], [77582529: 0/0], [77582525: 0/0], [77582521: 0/0], [77582517: 0/0], [77582513: 0/0], [77582509: 0/0], [77582508: 0/0], [77582506: 0/0], [77582505: 0/0],
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:30.465+0800 info
----------------------------------------------
JobId: 77582501,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: SAVE_META,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: 2023-08-25 17:48:24,
UploadFinishedTime: 2023-08-25 17:48:30,
FinishedTime: ,
UnfinishedTasks: ,
Progress: ,
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:32.476+0800 info
----------------------------------------------
JobId: 77582501,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: SAVE_META,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: 2023-08-25 17:48:24,
UploadFinishedTime: 2023-08-25 17:48:30,
FinishedTime: ,
UnfinishedTasks: ,
Progress: ,
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:34.490+0800 info
----------------------------------------------
JobId: 77582501,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: UPLOAD_INFO,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: 2023-08-25 17:48:24,
UploadFinishedTime: 2023-08-25 17:48:30,
FinishedTime: ,
UnfinishedTasks: ,
Progress: ,
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:36.501+0800 info
----------------------------------------------
JobId: 77582501,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: UPLOAD_INFO,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: 2023-08-25 17:48:24,
UploadFinishedTime: 2023-08-25 17:48:30,
FinishedTime: ,
UnfinishedTasks: ,
Progress: ,
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:38.512+0800 info
----------------------------------------------
JobId: 77582501,
SnapshotName: label_demo_ads_ec_20230825174817,
DbName: demo,
State: FINISHED,
BackupObjs: [demo.ads_ec],
CreateTime: 2023-08-25 17:48:18,
SnapshotFinishedTime: 2023-08-25 17:48:24,
UploadFinishedTime: 2023-08-25 17:48:30,
FinishedTime: 2023-08-25 17:48:37,
UnfinishedTasks: ,
Progress: ,
TaskErrMsg: ,
Status: [OK],
Timeout: 86400,
2023-08-25T17:48:40.513+0800 info 数据库快照备份完成!之前我也遇到个类似的问题, 不过我用的是腾讯的cos, 这个cos是一个融合桶, 所以StarRocks对文件改名的时候, 出现了没有权限的情况(这个当时再broker日志里面有记录, 可以看下), 但是后面我们解决的办法很简单, 换了一个普通的桶
嗯嗯,之前用oss有测过也是可以的,现在用GCS为存储桶不行了,后面我这测试下升级为3.x版本看下,看代码那2.5.6版本还不支持gc协议,3.x是有的