Support profile for AWS-based credentials for restic

[dinesh]

What steps did you take and what happened:

It seems Velero uses restic CLI under the hood for taking data backups/restores, though it runs fine for a single backup-location but not working if multiple storage-locations (s3 based) are present with different credentials, i.e. multiple AWS profiles in AWS_SHARED_CREDENTIALS_FILE file which Restic pod mounts at /credentials/cloud.

I guess the root of the problem is while executing restic command, Velero is not setting AWS_PROFILE which will help restic to load correct credentials.

What did you expect to happen:
Restore/Backup with Restic should work will multiple AWS credentials.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero Output
• velero restore describe r2 --details Output
• velero backup describe b1 Output
• velero backup logs b1 (outputs some junk)
• velero restore logs r2 (outputs some junk)
• velero restic repo get -ojson Output

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

As you can see above the restic repo is not getting in Ready state because of the credentials problem.

• velero get backup-locations -ojson Output

Note that it has two backuplocations with profile attribute set for testc1. Here is the full credentials secret -

• kubectl ksd -n velero get secrets cloud-credentials -ojson Output

Environment:

• Velero version (use velero version): v1.1.0
• Velero features (use velero client config get features): features: <NOT SET>
• Kubernetes version (use kubectl version): Client: v1.12.1 Server: v1.15.4-k3s.1
• Kubernetes installer & version:
• Cloud provider or hardware configuration: k3d/k3s
• OS (e.g. from /etc/os-release):

[dinesh]

@skriss is it a known thing or not yet implemented, I would be happy to contribute if you guys can give me some pointers.

[skriss]

@dinesh it would be great to have contribute here.

Couple notes:

• Velero (non-restic) does support different AWS profiles per BackupStorageLocation, by specifying the profile config key in your BSL (see https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/master/backupstoragelocation.md)
• It sounds right (though I haven't tested myself) that that code doesn't work when it comes to restic
• we have a similar need to set some environment variables when executing restic commands when running on Azure - see https://github.com/vmware-tanzu/velero/blob/master/pkg/restic/common.go#L221-L242 for the function that gets those, and you can grep the codebase to see the handful of places where that's used

Though we're not thrilled about having that provider-specific code inside Velero core and we'd eventually like to extract it, this is our current solution, and I think it'd be fine to follow a similar pattern for AWS, to add the AWS_PROFILE env var if needed when running restic commands.

Let me know if that's enough direction or if you could use some more pointers!

[dinesh]

@skriss makes sense, I guess I will dabble a bit in the code and try to create a fix for it.

[skriss]

@dinesh just checking in to see if you could use any further input here?

[dinesh]

@skriss I think I got the hang of it and raise a PR. I tested it in my local setup and it seems to work.

[skriss]

@dinesh great, we'll check out your PR shortly!