Forensic Container Checkpointing #2008
Assignees
Labels
sig/node
Categorizes an issue or PR as relevant to SIG Node.
stage/beta
Denotes an issue tracking an enhancement targeted for Beta status
Comments
k8s-ci-robot
added
the
needs-sig
|
/sig node |
k8s-ci-robot
added
sig/node
|
We recommend actively socializing your KEP with the appropriate sig to gain visibility, consensus and also for scheduling. Also as you are not sure of what SIG will sponsor this, reaching out to the SIGs to get clarity on that will be helpful to move your KEP forward. |
|
Hi @adrianreber Any updates on whether this will be included in 1.20? Enhancements Freeze is October 6th and by that time we require: The KEP must be merged in an implementable state Best, |
|
Hello @kikisdeliveryservice
Sorry, but how would I decide this? There has not been a lot of feedback on the corresponding KEP which makes it really difficult for me to answer that question. On the other hand, maybe the missing feedback is a good sign that it will take some more time. So probably this will not be included in 1.20. |
kikisdeliveryservice
added
tracked/no
|
Normally the sig would give a clear signal that it would be included. That would be by : reviewing the KEP, agreeing to the milestone proposals in the KEP etc.. I'd encourage you to keep in touch with them and start the 1.21 conversation early if this does not end up getting reviewed/merged properly by October 6th. Best, |
|
@kikisdeliveryservice Thanks for the guidance. Will do. |
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
the
lifecycle/stale
|
Stale issues rot after 30d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-testing, kubernetes/test-infra and/or fejta. |
k8s-ci-robot
added
lifecycle/rotten
|
Rotten issues close after 30d of inactivity. Send feedback to sig-contributor-experience at kubernetes/community. |
|
@fejta-bot: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
/reopen |
|
@adrianreber: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
removed
the
lifecycle/rotten
|
Issues go stale after 90d of inactivity. If this issue is safe to close now please do so with Send feedback to sig-contributor-experience at kubernetes/community. |
k8s-ci-robot
added
the
lifecycle/stale
|
/remove-lifecycle stale Still working on it. |
k8s-ci-robot
removed
the
lifecycle/stale
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
adrianreber
added a commit
to adrianreber/containerd
that referenced
this issue
Jul 16, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
adrianreber
added a commit
to adrianreber/containerd
that referenced
this issue
Jul 16, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
adrianreber
added a commit
to adrianreber/containerd
that referenced
this issue
Jul 18, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
adrianreber
added a commit
to adrianreber/containerd
that referenced
this issue
Jul 22, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
adrianreber
added a commit
to adrianreber/containerd
that referenced
this issue
Jul 23, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
adrianreber
added a commit
to adrianreber/containerd
that referenced
this issue
Jul 23, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
adrianreber
added a commit
to adrianreber/containerd
that referenced
this issue
Jul 23, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
adrianreber
added a commit
to adrianreber/containerd
that referenced
this issue
Jul 25, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
|
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
/remove-lifecycle stale |
k8s-ci-robot
removed
the
lifecycle/stale
|
Potential bug for beta: kubernetes/kubernetes#126232 This is actually by design. We don't want API control in this checkpoint case. |
|
@adrianreber What are your plans for this KEP for 1.32 (next month or two)? |
|
/assign @adrianreber |
|
As the KEP said, The Graduation Criteria includes containerd implementation containerd/containerd#6965 and containerd/containerd#6965 is included in containerd v2.0 which is not release yet. I'm afraid that we may wait for another release cycle unless containerd 2.0 will be released in a month. |
Interesting point of view. My point of view was that those criteria where both fulfilled. containerd has support as the PR you mentioned was merged. It is not part of a release yet, but I would have said merged is enough. The e2e tests are also working with CRI-O and containerd. I would say that those two items are fulfilled, but it seems it can be viewed differently.
I was planing on trying to move it from Beta to GA, but it seems it still needs some discussion if the given criteria for GA are fulfilled or not. I would say yes but @pacoxu has a different point of view. |
If we already build CI to test it with containerd latest code, it would be OK to be promoted to GA. |
rst0git
pushed a commit
to rst0git/containerd
that referenced
this issue
Oct 31, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
rst0git
pushed a commit
to rst0git/containerd
that referenced
this issue
Oct 31, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
rst0git
pushed a commit
to rst0git/containerd
that referenced
this issue
Nov 1, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
rst0git
pushed a commit
to rst0git/containerd
that referenced
this issue
Nov 4, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
rst0git
pushed a commit
to rst0git/containerd
that referenced
this issue
Nov 5, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
rst0git
pushed a commit
to rst0git/containerd
that referenced
this issue
Nov 6, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
rst0git
pushed a commit
to rst0git/containerd
that referenced
this issue
Nov 8, 2024
This implements container restore as described in: https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh The code changes are based on changes I have done in Podman around 2018 and CRI-O around 2020. The history behind restoring container via CRI/Kubernetes probably requires some explanation. The initial proposal to bring checkpoint/restore to Kubernetes was looking at pod checkpoint and restoring and the corresponding CRI changes. kubernetes-sigs/cri-tools#662 kubernetes/kubernetes#97194 After discussing this topic for about two years another approach was implemented as described in KEP-2008: kubernetes/enhancements#2008 "Forensic Container Checkpointing" allowed us to separate checkpointing from restoring. For the "Forensic Container Checkpointing" it is enough to create a checkpoint of the container. Restoring is not necessary as the analysis of the checkpoint archive can happen without restoring the container. While thinking about a way to restore a container it was by coincidence that we started to look into restoring containers in Kubernetes via Create and Start. The way it was done in CRI-O is to figure out during Create if the container image is a checkpoint image and if that is true we are using another code path. The same was implemented now with this change in containerd. With this change it is possible to restore the container from a checkpoint tar archive that is created during checkpointing via CRI. To restore a container via Kubernetes we convert the tar archive to an OCI image as described in the kubernetes.io blog post from above. Using this OCI image it is possible to restore a container in Kubernetes. At this point I think it should be doable to restore containers in CRI-O and containerd no matter if they have been created by containerd or CRI-O. The biggest difference is the container metadata and that can be adapted during restore. Open items: * It is not clear to me why restoring a container in containerd goes through task/Create(). But as the restore code already exists this change extended the existing code path to restore a container in task/Create() to also restore a container through the CRI via Create and Start. * Automatic image pulling. containerd does not pull images automatically if created via the CRI. There is an option in crictl to pull images before starting, but that uses the CRI image pull interface. It is still a separate pull and create operation. Restoring containers from an OCI image is a bit different. The checkpoint OCI image does not include the base image, but just a reference to the image (NAME@DIGEST). Using crictl with pulling will enable the pulling of the checkpoint image, but not of the base image the checkpoint is based on. So during preparation of the checkpoint containerd will automatically pull the base image, but I was not able how to pull an image blockingly in containerd. So there is a for loop waiting for the container image to appear in the internal store. I think this probably can be implemented better. Anyway, this is a first step towards container restored in Kubernetes when using containerd. Signed-off-by: Adrian Reber <areber@redhat.com>
|
Looks like containerd 2.0 is released : https://github.com/containerd/containerd/releases/tag/v2.0.0 |
|
containerd/containerd#10365 is still open |
That is not really needed for Forensic Container Checkpointing. That PR is just taking the functionality one step further. But not really covered by KEP #2008. It is mainly to offer users more functionality. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
and others
Enhancement Description
k/enhancements) update PR(s):k/k) update PR(s):k/website) update(s):k/enhancements) update PR(s):k/k) update PR(s):k/website) update(s):k/enhancements) update PR(s):k/k) update PR(s):k/website) update(s):The text was updated successfully, but these errors were encountered: