ADd addressttl to the proxy entries in shared, venue proxies

[mike-gangl]

A restart to the HTTPD proxies often fixes many issues. One issue might be that the load balancer a proxy is pointing to might change over time. If this happens, and our DNS TTL is infinite, it will eventually point to the wrong location. this may result in a 404 or a hung connection (gateway timeout).

For the proxy entries, we might want to set some parameters:
addressttl -1 TTL in seconds for how long DNS resolutions of the backend address are cached. -1 means until restart of Apache httpd.
This might help as restarting the proxy often fixes the issues, which is indicative of the above.

Slack Message

[mike-gangl]

The thought here is that the DNS caching might be causing issues with the application ELBs- restarts often 'fix' the issues, and it could be that a cached DNS entry is no longer working.

[galenatjpl]

One theory is that adding retry=5 disablereuse=On to end of apache2 ProxyPass line configs might solve this. We are experimenting with this..

[galenatjpl]

@mike-gangl I re-read your description above. I think you are suggesting setting something other than -1 for addressttl so that it can auto refresh without requiring a apache2 restart?

[mike-gangl]

One theory is that adding retry=5 disablereuse=On to end of apache2 ProxyPass line configs might solve this. We are experimenting with this..

We can certainly try this- it seems like it will re-query the DNS once by using disablereuse.

[mike-gangl]

@mike-gangl I re-read your description above. I think you are suggesting setting something other than -1 for addressttl so that it can auto refresh without requiring a apache2 restart?

i think the above command will work as well- disablereuse seems to be the nuclear option, where as addressttl will just requery the DNS ever Xttl, unless its set to -1 and then it never requiereis the DNS, which i think is the issue.

Regardless, we need to apply this in both the venue proxies and the Shared service proxies pointing at the venue proxies.

[jpl-btlunsfo]

I agree with Mike, in comparison to addressttl, disablereuse is kinda crazy as a fix- since it also disables connection sharing, putting more load/latency on both sides. It's kinda gross that -1 is the default value for addressttl, and that's probably the source of our issues to begin with.

I'm not sure what's smartest to replace it with? I feel like something non-zero (otherwise we're basically always requerying the DNS for every request; more load), but is 5s short enough to catch that DNS switch?

...Unfortunately I have a venue-proxy&MC currently stood up from another test, and was able to actually try it. It chokes with a:

AH00526: Syntax error on line 5 of /etc/apache2/sites-enabled/main.conf:
ProxyPass unknown Worker parameter

Poking around a bit the only explanation I can find is that our base image (ubuntu/apache2) is using apache HTTPD version 2.4.58:

$ docker pull ubuntu/apache2
Using default tag: latest
latest: Pulling from ubuntu/apache2
207a8499ffa9: Pull complete
1db32677b891: Pull complete
cbeb97bc6e2c: Pull complete
Digest: sha256:590b7b0f55fbfaf363be800d938247addfdfd461371082bc0cb56ac7dbc5876b
Status: Downloaded newer image for ubuntu/apache2:latest
docker.io/ubuntu/apache2:latest
$ docker run -it --rm ubuntu/apache2:latest bash
root@eb36e93eb4f1:/# dpkg -l | grep apache2\
ii  apache2                       2.4.58-1ubuntu8.4                 amd64        Apache HTTP Server

And based on this thread (I couldn't find any other source besides this), addressttl isn't available until 2.4.59.

[galenatjpl]

@jpl-btlunsfo did you do your poking around yesterday or so? I just redployed a new apache2 this morning, and it appears we are on 2.4.41:

$ dpkg -l  | grep apach
ii  apache2                              2.4.41-4ubuntu3.21                amd64        Apache HTTP Server

[galenatjpl]

@jpl-btlunsfo regardless, please do what you think is best in terms of the options, and apply this to both the venue proxies and the Shared service proxies pointing at the venue proxies.

[jpl-btlunsfo]

your poking around yesterday or so

No, that was from this morning (right before my message). The image/httpd version you're seeing is probably from the last time the httpd-proxy container was built, rather than my latest pull.

Regardless, neither seems to support addressttl yet., so I think disablereuse might be our only option until that situation changes/the image updates.

[jpl-btlunsfo]

So uh, had a thought and tested it, but is there a specific reason we're using the ubuntu/apache2 image? Is there an issue with the httpd (_/httpd) image that's preventing us from making use of that?

I ask because I was able to build our dockerfile (with minimal changes) pointing at that base image instead with no issue (and using 2.4.62). Should I throw that in a PR?