velero-io/velero

Restore logs are written to "read-only" backup storage locations

Open

#1,978 opened on Oct 18, 2019

View on GitHub
 (6 comments) (4 reactions) (1 assignee)Go (1,550 forks)auto 404
Help wantedIceboxReviewed Q2 2021kind/Bug

Repository metrics

Stars
 (10,111 stars)
PR merge metrics
 (PR metrics pending)

Description

Describe the problem/challenge you have I start up a new cluster which has a default backup storage location. I add a second read-only backup storage location to restore from. In IAM I want to give it only read permissions such as GetObject (we use multiple clusters and I want to be able to restore from a different cluster's bucket, but only give write permission to one cluster per backup bucket). But unfortunately when I run a restore it tries to write logs to the backup storage location that is read-only. It fails with this error in the logs:

time="2019-10-18T03:01:05Z" level=error msg="Error uploading restore results to backup storage" controller=restore error="rpc error: code = Unknown desc = error putting object restores/customerdata-2019-10-18-02-59-06-restore/restore-customerdata-2019-10-18-02-59-06-restore-results.gz: AccessDenied: Access Denied\n\tstatus code: 403, request id: D1CC034F0FC007C1, host id: RTA7KuHx/Ypg3WLvsubDvkMWglBuK1sKL+UnH9iIroSLAeTI1R75UC/CW4Eg4ydEg9/pbZd3W4k=" error.file="/go/src/github.com/heptio/velero/pkg/cloudprovider/aws/object_store.go:204" error.function="github.com/heptio/velero/pkg/cloudprovider/aws.(*ObjectStore).PutObject" logSource="pkg/controller/restore_controller.go:494"

This is pointing to this line in the code base: https://github.com/vmware-tanzu/velero/blob/master/pkg/controller/restore_controller.go#L479 (this is master, not sure why the line number isn't 494 as in the error, but it's the same error message).

Describe the solution you'd like Provide restore create command the same flag that backup create has: --storage-location string location in which to store the backup. Then I could point it at a read-only storage location but write the output to the storage location of my choosing. This way I could keep s3 write permissions on the bucket to only that cluster which needs it.

Anything else you would like to add: This backup storage location is defined as access mode ReadOnly. It feels like this is a violation of that configuration value or it's at least an unclear experience.

Environment:

  • Velero version (use velero version): 1.1.0
  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"2d3c76f9091b6bec110a5e63777c332469e0cba2", GitTreeState:"clean", BuildDate:"2019-08-19T12:36:28Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0", GitTreeState:"clean", BuildDate:"2019-06-06T01:36:19Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
  • Kubernetes installer & version: kubeadm same version as kubernetes cluster above: v1.14.3
  • Cloud provider or hardware configuration: aws
  • OS (e.g. from /etc/os-release): Amazon Linux 2

Contributor guide