Unable to do a production rolling/gradual upgrade from 1.x to 2.x

[vazkarvishal]

Describe the bug
the upgrade guide suggests completely removing the helm chart and re-installing the new one. This is impractical to do in a production system without causing an outage. How can we do a gradual upgrade?

To Reproduce
Try to upgrade from 1.x to 2.x

Expected behavior
Allow rolling updates (although CRD changes may be required). Ideally a version which supports both the contexts/both ways of running so people can "migrate" using feature flags/transition to the new ways, and then in subsequent minor versions remove support for the old version/ways?

Additional context
If this is not allowed, it will be severly disruptive to take our entire production environment down everytime we have a major version upgrade. Can we have a better upgrade progress? We are currently stumped because of this issue.

[nginx-bot]

Hi @vazkarvishal!

Thanks for opening this issue!
Be sure to check out our Contributing Guidelines and the Issue Lifecycle while you wait for someone on the team to take a look at this.

[sjberman]

@vazkarvishal There are two hurdles here. One, the NginxProxy CRD changed from a cluster-scoped CRD to a namespace-scoped CRD, so it requires a delete and re-apply. You may be able to update this resource on its own and then do a normal helm upgrade or deployment rollout to see if this is a smoother process for you.

However I can't guarantee that there won't be slight downtime due to the change in architecture (which is the second hurdle). NGINX has been split out into its own Deployment and Service, and has to be provisioned. With such a large change in architecture from 1.x to 2.x, we weren't going to support both ways in the same version release, due to the logistical nightmare that could cause in the code. We essentially rewrote a large portion of our control plane to support this new architecture.

Ideally, we are going to try our best to avoid major version bumps when possible to avoid these types of disruptions. But a major version change does mean that it is not backwards compatible, and therefore the upgrade process will require more care than a normal minor/patch version change.

Another option that may work is installing the new version in a different namespace, and migrating your Gateway(s) (or duplicating them) to point at the new GatewayClass, before uninstalling the old version. Though I have not tested this method myself.

[vazkarvishal]

Ohh :( okay I can try the options you mentioned and get back to you. In the meanwhile, what are the plans for v1.x? Can we push a fix which allows setting requests and limits to the nginx container in 1.x?

I highly doubt I can convince my folks to have an outage based swap in prod.

[sjberman]

What do you mean by setting requests and limits? Right now we don't have any plans to add features or new functionality to v1.x.

I confirmed that you can run v1.x and v2.x at the same time in different namespaces (though you do still have to uninstall the old NginxProxy CRD before installing the new one, so the v1.x controller won't be able to read that config once the update occurs). You could probably create a new Gateway so that the new nginx instance gets provisioned by v2, adjust the Routes to point at the new Gateway (or duplicate the Routes to leave the old ones intact), and then DNS swap to the new nginx Service IP address when it's ready. I think this would reduce the downtime (or possibly eliminate it altogether).