[Feature]: Zero-downtime in-place update for services

[peterschmidt85]

Problem

Currently, when re-deploying a service (e.g., a newer version), dstack stops the existing run before submitting the new one. This causes a disruption, which is unacceptable for any production-grade service.

Solution

1. Introduce a revision number for service replicas.
2. During an in-place update, ensure that older replicas are only stopped once the newer replicas are successfully running.

Workaround

No response

Would you like to help us implement this feature by sending a PR?

Yes

[jvstme]

There is a hacky workaround to achieve zero-downtime deployment. It works if there are only changes in the repo and not in the service configuration.

1. Add at least one extra replica in the replicas configuration property and reapply the configuration. This will upload the new repo revision to dstack and start a new replica/replicas to handle traffic while the other replicas are being redeployed.
2. Manually stop the old replicas by making your application exit cleanly. For example, attach to a replica with dstack attach --replica <replica num> <run name>, SSH into the container, and send a signal to your application. This will force dstack to redeploy the stopped replicas using the new repo revision.
3. Remove the redundant replica(s) from the replicas configuration property and reapply the configuration.

This workaround is very limited and inconvenient, so we are interested in supporting proper zero-downtime deployments orchestrated by dstack.

[movchan74]

Hi @jvstme

I've tried the workaround, but I ran into some issues with scaling (adding new replicas).

I've tried to scale from 1 replica to 2. When my app was starting, the API became unstable, and ~50% of requests were failing because they were directed to the replica where the app was not ready yet. I got 404 because my routes were not added yet.

It takes a few minutes to start my app because it has to load all the models on the GPU, etc. So I cannot use scaling because it degrades the service.

Maybe we can add a health check in nginx to make sure that the requests are not sent to the broken replica?

This is the issue not only for this workaround but for the scaling in general.

[jvstme]

Hi @movchan74

I think Nginx has built-in availability checks enabled by default, but they only consider the replica unavailable if Nginx can't communicate with it at all, e.g. if the app is not bound to the port. So if you could adjust your app to only bind to the port once it's ready, it should work.

But many apps can't be implemented that way, so I agree that we'll need to support health checks. We intend to support them natively in #2181. I think this will be a prerequisite for the current ticket

[movchan74]

Hi @jvstme, thanks for the pointers so far!

I’ve managed to get around Nginx routing to unready replicas by proxying 127.0.0.1:8001 → 8000 with socat, so the new replica only appears healthy once it’s fully up. Unfortunately, I still can’t get the zero-downtime update to actually pick up my new code:

• Scaling the run (e.g. from 1→2 replicas) keeps using the old repo_hash.

• Manually bumping repo_hash in the config run fails with

  Cannot override active run. Stop the run first.
  

It looks like runs are immutable by design, so I can’t do an in-place code swap. Do you have any suggestions for safely rolling out a new commit with zero downtime, or should we try something else entirely until #2180 lands?

Here’s the script I’m using to scale runs for reference:
https://gist.github.com/movchan74/4a3799dbd5577af0ab98320f1eb7402f

Thanks!

[jvstme]

@movchan74, indeed, I've tested it again and found that only the uncommitted diff (repo_code_hash) can be updated without stopping the run. Deploying a new commit (repo_hash) currently requires stopping the run.

So far, I can't suggest any other workarounds. Then we'll try to prioritize #2180 so that it lands sooner

[movchan74]

Hi @jvstme, thanks for confirming my suspicion. I’ll hang tight for #2180. I appreciate you prioritizing this!

[jvstme]

@movchan74, I believe you mentioned blue-green deployments before. In the future, we can support different deployment strategies, including blue-green. However, we are considering starting with rolling deployments first.

Service replicas will be updated one by one to avoid using too many instances at once. The suggested algorithm is:

1. Start a new replica.
2. Wait until the new replica becomes running or healthy (once we implement [Feature]: Allow to configure a service healthcheck #2181).
3. Stop the old replica.
4. If there are more out-of-date replicas, go to step 1 for the next replica.

If the service only has one replica, this strategy is similar to blue-green deployments, except there can be a short period of time (a few seconds) when the new and the old replica coexist and both handle requests.

Will this work for you?