Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore fails with error "no space left on device" if PVCs are 100% utilised during backup #2812

Open
nrb opened this issue Aug 11, 2020 · 1 comment
Labels
Bug Restic - GA needed for restic integration to be considered GA Reviewed Q2 2021

Comments

@nrb
Copy link
Contributor

nrb commented Aug 11, 2020

What steps did you take and what happened:
From an internal bug report:

Using Restic, restore is failing with error "no space left on device" if PVC was fully occupied by the user data while the backup is taken.

for example, PVC of size 1Gi and user data in the PVC is ~upto 100% which is of 1Gi

What did you expect to happen:

Velero restic restores should be successful, even if the PV is full.

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

root@k8-master-277:~# velero restore describe restore001 --details
Name:         restore001
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:  PartiallyFailed (run 'velero restore logs restore001' for more information)

Errors:
  Velero:   pod volume restore failed: error restoring volume: error creating .velero directory for done file: mkdir /host_pods/60880bf2-9d1c-47cf-ba8f-5f8db43b385b/volumes/kubernetes.io~csi/pvc-ee79e0f0-ed62-44f7-987e-16efca1d1cd5/mount/.velero: no space left on device
  Cluster:    <none>
  Namespaces: <none>

Anything else you would like to add:

Since this is related to creating the .velero donefile, the fix might be somewhat relate to #2722.

Environment:

  • Velero version (use velero version): v1.4.0
  • Cloud provider or hardware configuration: vSphere

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

  • 👍 for "I would like to see this bug fixed as soon as possible"
  • 👎 for "There are more important bugs to focus on right now"
@nrb nrb added Bug P1 - Important Restic - GA needed for restic integration to be considered GA labels Aug 11, 2020
@nrb
Copy link
Contributor Author

nrb commented Sep 15, 2020

Scoping this for v1.6 currently; I'm unsure if the fix will be backport-able at this time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Restic - GA needed for restic integration to be considered GA Reviewed Q2 2021
Projects
None yet
Development

No branches or pull requests

3 participants