Is incoming traffic allowed to my application during mongock changes

[vallishk]

Hi All,
My springboot application is running on EKS with multiple instances and a rolling update is being applied. The update contains database changes (large amount of data migration/movement and some index updates). While this change is being applied, will incoming traffic requests still be handled by this new instance? Can I assume my springboot application as being fully up and will health checks pass during this time?

[dieppa]

Mongock is first designed to prevent the application to be deployed until the migration is finished, which means the inbound traffic won't be handled. However, there are ways to alter this behaviour, for example in Springboot, you can ply with ApplicationRunner and InitializingBean. Also in K8S you have the probes liveness, readiness and startup, which you can configure as you wish to allow/prohibit your inbound traffic.

At this point, when designing your changeUnits, you should take into account the HA, which can force you to split your change into 2(or even 3) steps in order to prevent an inconsistent state in your database.

[vallishk]

Thank you for the explanation. 2 follow-up questions arise:

1. Does this mean that in the scenario where multiple replicas of the service are deployed, it is possible that replica 1 is applying the change while replica 2 may be handling traffic (and is live) and is probably reading/writing to the same database?
2. Does replica 2 not attempt to apply the database changes that are being rolled out at some point of time? Once the lock is released by replica 1 (which has applied a new change unit successfully), would replica 2 also verify if something has to be applied? If yes, until such time, how would replica 2 be considered as deployed, up and running?

[dieppa]

Hello @vallishk ,

before answering, I will explain how rolling updates work in Kubernetes(the most common and default deployment strategy in K8s). You probably already know this, but just to make sure we are in the same page:

This strategy aims to update the workload without downtime, by incrementally replacing the old pods with the new ones.

Let’s imagine a scenario where your deployment(old and new) has 3 replicas(R1, R2 and R3). When performing a rolling update, Kubernetes will start one new pod, which will apply the DB changes. After succeeding the startup and liveness probes(in this order), it checks the readiness probe to see if the new pod is ready to receive traffic and if so, replace one of the old pods with this new one. This process will be performed on the remaining pods, until all the pods in the "old" deployment are replaced by the new ones.

Two things to take in mind.

1. Although Mongock will check in every deployment if there are missing changeUnits to apply, only the first one will apply them, as by the time the other deployments check the Mongock's state, the changeUnits are already applied
2. Even if the replicas are deployed concurrently, Mongock uses a sophisticated distributed lock that will prevent from any race condition.

Lets now answer your questions:

1. Yes, in the context of High Availability, it means that. If you can afford such a behaviour, you can always do a "stop the world" to perform the deployment, which means stopping the service while you perform your deployment. But as you probably know, this is a poor strategy, it normally provides a bad user experience and rarely a modern service can afford it. As I mentioned the other day, for this kind of change we suggest a multiphase migrations, which consist in splitting the migration in smaller intermediate pieces that allows two consecutive versions to coexist. Sometimes, depending on the complexity of the deployment, it requires more than two versions of the application to be compatible. For example, if you want to rename a field, the first deployment add a new field with the new name, and copy the values. At the end of the deployment you have the two fields with the values duplicated, but now you are in a good position to do another deployment which delete the old field.

2. Mongock will always check if there is any change to apply. If there is any pending change to apply, it acquires the lock, to avoid any race condition, and perform the migration. Otherwise just finish the Mongock process without even taking the lock.

[vallishk]

Thank you!
That makes it clear.

[rahulsant100]

@dieppa Point no. 2 is a bit unclear. The question is incase of concurrent deployment (For eg: 3 pods with mongock enabled are deployed concurrently), let's suppose pod1 acquires the lock and runs the pending migrations. Till the time pod1 is running the migration, Will pod2 and pod3 wait for pod1 to finish the migration and application won't start in pod2 and pod3 ?

[dieppa]

Exactly, pod1 will acquire the lock, while the other will wait. During that time you may experience some pod restarts for pod2 and 3, depending how long the migration takes and your Kubernetes configuration. Anyway, eventually, once the Mongock process is finished, and the lock is released, pod 2 and 3 will check the if there are pending changes again, as there isn't any, they won't even try to acquire the lock, and will be successfully deployed.