Snapshot / Restore Volume Support for Kubernetes (CRD + External Controller)

[jingxu97]

Feature Description

• One-line feature description (can be used as a release note):

Snapshot / restore functionality for Kubernetes and CSI. This provides standardized APIs design (CRDs) and adds PV snapshot/restore support for CSI volume drivers.

Old description:

Expose the ability in the Kubernetes API to create,list, delete, and restore snapshots from an arbitrary underlying storage systems that support it.

• Primary contact (assignee): @jingxu97, @xing-yang
• Responsible SIGs: storage-SIG
• Design proposal link (community repo):
  • Old: CSI snapshot design community#2335
  • Old: https://docs.google.com/document/d/17WS4Wk4MXRH24i-BpMpIFo5F-SNoRkm_KtkBMZEEoAo/edit
• KEP: https://github.com/kubernetes/enhancements/tree/master/keps/sig-storage/177-volume-snapshot
• Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Luis Pabón (@lpabon)
  • Ritesh Shukla (@kerneltime)
  • Brad Childs (@childsb)
• Approver (likely from SIG/area to which feature belongs):
  • Saad Ali (@saad-ali)
• Feature target (which target equals to which milestone):
  • External repo (x.y): 1.8
  • Alpha release target (x.y): 1.12
  • Beta release target (x.y): 1.16

[davidopp]

kubernetes/kubernetes#39657
kubernetes/kubernetes#44172

[timothysc]

/cc @skriss

[mdelio]

@jingxu97 I think we're going to have something in alpha for 1.7, can we can we please set the milestone to 1.7?

[calebamiles]

@kubernetes/sig-storage-feature-requests could someone please update the issue description to the new template. Thanks!

[jingxu97]

I will work on it. Thanks! Best, Jing

…

On Thu, May 4, 2017 at 8:43 AM, caleb miles ***@***.***> wrote: @kubernetes/sig-storage-feature-requests <https://github.com/orgs/kubernetes/teams/sig-storage-feature-requests> could someone please update the issue description to the new template. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#177 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ASSNxR4fUmmxnJ4QQc-Fe0K5CKT5tRO9ks5r2fIcgaJpZM4Lrsji> .

-- - Jing

[alkar]

Is this based on the proposal here: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/volume-snapshotting.md ?

[jingxu97]

No, the new proposal doc is here https://docs.google.com/document/d/17WS4Wk4MXRH24i-BpMpIFo5F-SNoRkm_KtkBMZEEoAo Please let me know if you cannot open it. Thanks! Jing

…

On Tue, May 9, 2017 at 7:34 AM, Dimitrios Karagiannis < ***@***.***> wrote: Is this based on the proposal here: https://github.com/kubernetes/ community/blob/master/contributors/design-proposals/volume-snapshotting.md ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#177 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ASSNxVKLqOq9lsJ72MrdqkNN-2pbfmtIks5r4HmQgaJpZM4Lrsji> .

-- - Jing

[alkar]

@jingxu97 thanks! I can't access it but I just requested access.

[yanivlavi]

@jingxu97 I wanted to add an important note here regarding overall approach. I'm not this is the right place to put this, but I would be a happy to be guided to the right forum to bring this up.

Coming from a long background in oVirt and OSP I feel there is an important aspect that needs to be discuss which is ownership of the state of snapshots (and overall volume metadata) and discoverablility of the this metadata.

In cloud and on premise users (developers and admins) might prefer the native storage APIs over the Kube APIs. Kube is also not the one creating the snapshot, it is the cloud service or the storage itself, therefore it is not the owner of this metadata.

In OSP they have already made this mistake with Cinder that forces users that want to use snapshots to go through the Cinder API to have it be available in the OSP environment. Cinder doesn't know snapshot created on the storage and if you lose the Cinder you lose everything as the metadata that counts is in his stateful DB.

What I'm trying to say is that it is important that Kube doesn't try to be the owner of the volume metadata, which means things like periodically checking if a new snapshot was created directly via the storage service API or that the ID used for the snapshot is the storage service snapshot ID.

It is clear that Kube should expose snapshotting since it very needed to the container use case, but it is very important it doesn't become a storage abstraction service like Cinder. We do not want to chase the storage service features or limit user to use the Kube API, it should be a option. We also want to allow discovery of volumes no matter where they where created.

As we get to more complex features like QOS and oversubscription for example, we want to allow exposing and reusing the storage service capabilities, not replace them or block users from using them via the cloud service or storage management APIs.

[idvoretskyi]

@jingxu97 any progress on the feature description? @kubernetes/sig-storage-feature-requests

[idvoretskyi]

@mdelio @jingxu97 please, update the feature description with the new template - https://github.com/kubernetes/features/blob/master/ISSUE_TEMPLATE.md

[saad-ali]

@idvoretskyi I updated the original comment with the new template. This feature is not actually shipping any bits in the Kubernetes core for v1.7, so therefore I moved it to the next-milestone. This means that it will not need documentation, etc. for 1.7 in the Kubernetes core. I will remove the feature from the 1.7 tracking board as well.

[idvoretskyi]

@saad-ali any updates for 1.8? Is this feature still on track for the release?

[rootfs]

cc @tsmetana

[childsb]

@idvoretskyi this is still on track: kubernetes-retired/external-storage#331

[jdumars]

@jingxu97 @rootfs any update on missing docs for this? PR is due today.

[xing-yang]

Thanks @eagleusb ! I'll submit a placeholder doc soon.

[eagleusb]

Hi @xing-yang 👋

Thanks for your update. In the meantime, the docs placeholder deadline is almost here.

Please make sure to create a placeholder PR against the dev-1.20 branch in the k/website before the deadline.

Also, please keep in mind the important upcoming dates:

• Friday, Nov 6th: Week 8 - Docs Placeholder PR deadline
• Thursday, Nov 12th: Week 9 - Code Freeze
• Monday, Nov 23rd: Week 11 - Docs PR Ready for ReviewAs a reminder, please link all of your k/k PR as well as docs PR to this issue for the release team to track.

[xing-yang]

Hi @eagleusb ,

Thanks for the reminder! Doc PR is submitted here: kubernetes/website#24849

[kikisdeliveryservice]

Hi @xing-yang

Looks like kubernetes/kubernetes#95282 is still open but being actively worked on. Just a reminder that Code Freeze is coming up in 2 days on Thursday, November 12th. All PRs must be merged by that date, otherwise an Exception is required.

Best,
Kirsten

[xing-yang]

Hi @kikisdeliveryservice ,

Thanks for the reminder! We are trying to get reviewers to finish reviewing and approving the PR by the 11/12 deadline.

Xing

[xing-yang]

This PR that updates snapshot CRDs to v1 for cluster addon is merged: kubernetes/kubernetes#96383
We are moving closer.

[kikisdeliveryservice]

Great! just waiting on kubernetes/kubernetes#95282

[xing-yang]

@kikisdeliveryservice ,

kubernetes/kubernetes#95282 is approved. Just waiting for it to be merged:).

[kikisdeliveryservice]

Yay! It's merged! Updating tracking sheet.

Congrats! 🎆

[xing-yang]

Thanks @kikisdeliveryservice!

[kikisdeliveryservice]

Hi @xing-yang

Can you update the kep.yaml to reflect a status of implemented:

enhancements/keps/sig-storage/177-volume-snapshot/kep.yaml

Line 10 in f40c24a

status: implementable

Once that merges we can then close this issue.

Thanks!
Kirsten

[xing-yang]

Hi @kikisdeliveryservice,

Will submit a PR soon. Thanks!

Xing

[kikisdeliveryservice]

Thanks @xing-yang it's merged! Feel free to close this issue 😄

[annajung]

Hello, 1.21 Enhancement lead here.
I'm closing out this issue since the enhancement is GA and KEP has been updated to implemented.

/close

[k8s-ci-robot]

@annajung: Closing this issue.

In response to this:

Hello, 1.21 Enhancement lead here.
I'm closing out this issue since the enhancement is GA and KEP has been updated to implemented.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[mickeyboxell]

Hi @jingxu97 👋 1.24 RT Comms lead here. I saw a note that the VolumeSnapshot v1beta1 CRD will be removed in 1.24. Would this be appropriate to include in our 1.24 Removals and Deprecations blog post?

[jingxu97]

Yes, I think so @mickeyboxell

@xing-yang

[mickeyboxell]

Thanks for confirming! @jingxu97 What information would you like communicated in the blog? I read that the functionality entered beta in 1.20 and was a little confused about the v1beta1 CRD now being removed. Did the project graduate to stable or was it replaced with an alternative API?

[jingxu97]

VolumeSnapshot went GA in 1.20
#1994
https://kubernetes.io/blog/2020/12/10/kubernetes-1.20-volume-snapshot-moves-to-ga/

[xing-yang]

@mickeyboxell I added that entry in the spreadsheet. VolumeSnapshot went GA in 1.20. Following K8s 1.21 release, we deprecated VolumeSnapshot v1beta1. Since VolumeSnapshot is out-of-tree CRD, we have the deprecation message in the release note here:
https://github.com/kubernetes-csi/external-snapshotter/releases/tag/v4.1.0

Now we are ready to remove VolumeSnapshot v1beta1 CRD in our next external-snapshotter release which will be v6.0, shortly after K8s 1.24 release.

We want to add a message in the deprecation/removal blog to indicate that VolumeSnapshot v1beta1 CRD will be removed in K8s 1.24.

Hope this helps.

[xing-yang]

@mickeyboxell In addition to the deprecation/removal blog, I'd also like to have a release note in K8s v1.24 release notes to indicate VolumeSnapshot v1beta1 CRD will be removed. Where can I add that release note?

[mickeyboxell]

I'm not sure how their process works. You may want to reach out to the #release-notes channel for more information.