Ability to set upstream zone size and keepalive settings

[kate-osborn]

As a user of NGF
I want NGF to update the upstream zone size for NGINX
So that if I run into errors when due to exceeding my zone size, I can fix them.

As a user of NGF
I want NGF to enable keepalive connections on my route
So that I can optimize the performance of my application.

Acceptance

• The user is able to set NGINX's upstream zone size.
• The user is able to enable keepalive connections as defined by the design.
• When possible, configuration updates with NGINX Plus should be made using the NGINX Plus API so NGINX is not reloaded.
  • zone size
  • keepalives connections

Dev Notes:

• If a location block is forwarding to an upstream with keepalives, the Connection header must be empty. See http://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive

### Tasks
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/2809
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/2810
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/2811
- [ ] https://github.com/nginxinc/nginx-gateway-fabric/issues/2812

[mpstefan]

Would the user expect the system to do this? What is the impact on the user's system if the zone size is not dynamically updated?

[brianehlert]

Dynamic calculation of upstream zone accomplishes one thing, as the size of an upstream service grows it ensure that NGINX can handle that.
The positive impact is that the customer can scale upstream services at will to 1000s of pods and the system will dynamically adapt.
The negative impact is the growth in memory utilization as the possibility exists to high limits.

I have an entire write-up around this for NIC to do the same.
This would be a later and advanced capability that has the impact of optimizing the system.

If not dynamically calculated, it needs to be exposed in a configmap for example or however system tuning is exposed.
Whether auto-magic is a requirement for v1 should be discussed.

[mpstefan]

Today we discussed on how this would be valuable for NGINX+ as it does not require a reload when upstreams are added or removed.

[brianehlert]

NIC has run into a number of customer situations where customers set their limits so lean as even a back end service scaling event can cause OOM or CPUThrottling situations as a result of the configuration change. Without conscious memory consumption increases that are introduced by a feature like this.

While I think this capability is highly valuable,
As I have learned more about how customers are leaning into using Quality of Service and other K8s platform requirements that force the setting of limits - I am hesitant at introducing something like this due to the feared impact on the system as a whole and a situation where the Gateway is unable to start because the configuration alone is forcing the pod into an OOM state.