How to access old objects ?

[NomadXD]

For example let's say I have a resource and then I update it. I get the reconcile request with name and namespace.

I can access the object with r.Get() in the reconciler.

But how to get the previous state of that object ?

[camilamacedo86]

In the context of Kubernetes and controllers built using kubebuilder, when an object is updated, the reconciler is triggered with the current state of the object. The previous state of the object is not automatically provided.

You might find ways to do so, by for example storing the old state. Not sure if that would be an ideal approach.

[NomadXD]

@camilamacedo86 Thanks for the reply and this was opened last year. What I did was store the objects in the operator and compare the generation field whenever a reconcile happens to know whether the operator has the latest version or not. Works well for my use case without any issue. Thanks for the response.

[camilamacedo86]

Hi @NomadXD,

The core question is, why is there a need to be aware of the previous state of the resource? It's essential to understand that the fundamental design principle of Kubernetes is to ensure that the desired state aligns with the current state. If you could shed some light on the use case or explain why this requirement exists in the first place, it would provide better clarity.

[NomadXD]

@camilamacedo86 I understand the fundamental design of the controllers with control loop to match the current state with the desired state. But in a software application development perspective, there could be scenarios where the desired state cannot be achieved and rollback to previous desired state (last working state) is required. (Deployment rollbacks in K8s)

For our use case we wanted to rollback to a previous API revision deployed in an API gateway automatically if the API contract tests are failed. This can be done by storing the last working API Custom Resource and applying manually. This would increment the generation field by 1 and adds a "machine" generated API revision. Ideally we wanted to revert back to the last working "human applied" API revision.

[camilamacedo86]

Hi @NomadXD,

If you want to "try out" a Kubernetes manifest to see what changes it would make to your cluster without actually applying them, you can use the --dry-run option with kubectl apply.

This lets you simulate the application of the manifest and verify its correctness without making actual changes to your cluster resources. Also, why not check the conditions before applying the change? Can you not validate the required circumstances?

However, the concept of "rolling back to a previous desired state" can be thought of in two ways:

• Implicit Rollback: In the event that a resource (e.g., a Pod) cannot achieve its desired state, Kubernetes controllers will continually try to reconcile until it matches the desired state or until manual intervention. This process doesn't implicitly roll back to a previous state but tries to achieve the specified state.
• Explicit Rollback: Deployment rollbacks in Kubernetes, as you mentioned, are an explicit action where the desired state is intentionally set to a previous known good configuration. This is not the system reverting on its own but rather a user-driven action.

Therefore, It might be a result of more bad design choices which is not aligned with k8s itself than a real need. I would like to highlight some thoughts:

• Declarative Nature: Kubernetes is built on the philosophy of declaring desired states for resources. By diverging from this declarative approach and implementing mechanisms that track and possibly revert to past states, we might complicate the inherent simplicity and predictability that Kubernetes offers.
• Idempotency and Reconciliation: Kubernetes controllers are designed to be idempotent and continuously reconcile to ensure the actual state matches the desired state. By introducing logic that can change the desired state based on past information, we risk introducing non-idempotent behaviors and complex reconciliation logic.
• State Management: Storing previous states of a resource adds an overhead of managing and tracking that state, leading to increased complexity. This approach can also raise questions about how long we store past states, how many past states we store, and how we handle conflicts between past and present states.
• Error Recovery: While it's essential to have mechanisms for error recovery, it might be more prudent to build robust monitoring, logging, and alerting solutions that notify us of discrepancies, rather than automatically reverting to past states. This way, we can ensure that we're addressing the root causes of problems rather than masking them.

I hope that helps you out.

[NomadXD]

@camilamacedo86

then k8s allow us use try-out options
So if you find an issue you do not move forward

What try-out options you mean here ? K8s dry run ? Dry run only allows us to check whether the Spec is valid or not.

So, why not check the conditions before to apply?

To check that before deploying to the production environment, a sophisticated deployment strategy like K8s blue/green deployments are required and that is expensive in terms of resources. Also no matter how hard we test in pre-production environment, still it can break in production environment due to configuration differences or any 3rd party external dependencies. If we can check everything before applying, we wouldn't need K8s rollbacks at all for K8s deployments. Having the ability to rollback to previous versions is a valid use case in many scenarios.

[camilamacedo86]

Hi @NomadXD

generation field whenever a reconcile happens to know whether the operator has the latest version or not

It is a good practice to fetch the resource before updating or changing it, see the code generated when we are using the Deploy Image plugin which will add the reconciliation code for you: (so that you avoid facing the error: `the object has been modified, please apply // your changes to the latest version and try again")

https://github.com/kubernetes-sigs/kubebuilder/blame/42c5220993b4b2dcd5b1fedf034f4964807561db/testdata/project-v4-with-deploy-image/internal/controller/busybox_controller.go#L109-L117

Also, note that K8s stores the old version of the resource in the annotations.
However, the whole goal is to declare your desired state and the controller/reconciliation will be in a loop until it is achieved.

It seems that this one is sorted out so I am closing this discussion topic
But if you think that you need help within please feel free to re-open.