php:7.2-apache unusable on AWS ECS with application load balancer due to SIGWINCH

[hburrows]

Environment:

• Amazon Elastic Container Service using EC2 (not Fargate)
• Container configured to use Application Load balancer

When trying to use php:7.2-apache (via mediawiki derived image) it immediately fails with caught SIGWINCH, shutting down gracefully when added to the load balancer or on first health check. No access requests ever appear in the log so I suspect the issue might happen prior to the initial health check-- plus I can simulate the health check when not connected to the load balancer and everything is fine.

When I run a container using the same Task Definition not connected to a load balancer it works fine.

In this environment using docker run -d ... to run in daemon mode is not an option.

Is there another way to start apache2 which will not run in the foreground and eliminate this error?

This is what's in the logs (via docker logs <<id>>)

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
[Fri Sep 13 15:29:05.496367 2019] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.2.22 configured -- resuming normal operations
[Fri Sep 13 15:29:05.497503 2019] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
[Fri Sep 13 15:31:40.422800 2019] [mpm_prefork:notice] [pid 1] AH00170: caught SIGWINCH, shutting down gracefully

Sometimes the logs are completely empty. There is never any access request logged suggesting that the issue happens before the initial health check or as part of handling the health check before logging it.

NOTE: I'm using this environment because 1) the application load balancer terminates SSL, 2) it's organizationally efficient/consistent from an ops perspective to have everything dockerized and running in ECS via EC2 or Fargate.

[wglambert]

So given that you're running the service in the foreground when encountering this issue then unfortunately we've accurately reproduced the expected behavior from upstream https://bz.apache.org/bugzilla/show_bug.cgi?id=50669

You are correct that SIGWINCH is used to signal a windows size change to the applications running on a terminal. Due to the lack of other signal numbers and given the fact that httpd normally runs detached from a terminal the SIGWINCH signal was chosen to instruct httpd to do a graceful shutdown. So this works as designed.

[wglambert]

Even changing the STOPSIGNAL we've just added won't alleviate this, as when running in the foreground Docker will pass a SIGWINCH regardless

$ docker build -t php:test - << EOF                                                                             
FROM php:7.2-apache                                                                                                                    
STOPSIGNAL SIGTERM                                                                                                                     
EOF                                                                                                                                    
                                                                                                                                       
Sending build context to Docker daemon  2.048kB                                                                                        
Step 1/2 : FROM php:7.2-apache                                                                                                         
 ---> 35e5c60e1f7d                                                                                                                     
Step 2/2 : STOPSIGNAL SIGTERM
 ---> Running in e0eb6f47fa50
Removing intermediate container e0eb6f47fa50
 ---> 3999189c0c68
Successfully built 3999189c0c68
Successfully tagged php:test

$ docker run --name php php:test
AH00558: apache2: Could not reliably determine the server's fully qualified dmain name, using 172.17.0.2. Set the 'ServerName' directive globally to supprss this message
AH00558: apache2: Could not reliably determine the server's fully qualified dmain name, using 172.17.0.2. Set the 'ServerName' directive globally to supprss this message
[Fri Sep 13 16:54:18.528976 2019] [mpm_prefork:notice] [pid 1] AH00163: Apach/2.4.38 (Debian) PHP/7.2.22 configured -- resuming normal operations
[Fri Sep 13 16:54:18.529016 2019] [core:notice] [pid 1] AH00094: Command line 'apache2 -D FOREGROUND'

<resize window>

[Fri Sep 13 16:54:21.432058 2019] [mpm_prefork:notice] [pid 1] AH00170: caught SIGWINCH, shutting down gracefully

[hburrows]

@wglambert Thanks for the fast response. I've done a ton of googling regarding this and have concluded the current behavior is expected. What I'm really looking for is an alternative way of running apache to avoid this issue. This seems very crippling especially for folks trying to run in AWS/ECS (which I can't imagine I'm alone in doing... but maybe).

Can you suggest another way of running apache? Should I just define my own CMD ["/etc/init.d/apache2 start"] or something like that? Why do the current configs run Apache in the foreground? What is the advantage of that?

That current setup flat out doesn't work on AWS/ECS behind an application load balancer. Would you suggest I try a different stack using nginix?

Any suggestions to how to move forward would be greatly appreciated. Thanks!!

[wglambert]

So if you do --sig-proxy=false then Docker won't proxy signals to the process
https://docs.docker.com/engine/reference/commandline/attach/

[hburrows]

@wglambert That isn't an option when running with AWS/ECS (as far as I have been able to determine). Seems like they should support all the options (or at least provide a mechanism to express them) but that isn't the case, sadly.

[hburrows]

@wglambert Your proposal using attach definitely makes sense... but I don't see how to implement it in ECS. With ECS, you define a Task Definition, create a container using that Task Definition, and ECS tasks care of the rest. Once it's running I can determine which instance it's running on, SSH to that instance, and then attach but I see no what of automating that flow.

[wglambert]

attach is what you're in when running the container in the foreground

$ docker run --rm --sig-proxy=false --name php php:7.2-apache
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
[Fri Sep 13 18:02:25.103511 2019] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.2.22 configured -- resuming normal operations
[Fri Sep 13 18:02:25.103623 2019] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'

<resize window>
. . .

<send ctrl+c>
^C

You can also change the CMD to use the shell form then it will run in a subshell, which won't proxy signals.
https://docs.docker.com/engine/reference/builder/#cmd

$ docker build -t php:test - << EOF
FROM php:7.2-apache                                                              
CMD apache2-foreground
EOF

Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM php:7.2-apache
 ---> 35e5c60e1f7d
Step 2/2 : CMD apache2-foreground
 ---> Running in 2c3503139539
Removing intermediate container 2c3503139539
 ---> 37ce66a5f66e
Successfully built 37ce66a5f66e
Successfully tagged php:test

$ docker run --rm --name php php:test
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
[Fri Sep 13 17:57:25.863012 2019] [mpm_prefork:notice] [pid 6] AH00163: Apache/2.4.38 (Debian) PHP/7.2.22 configured -- resuming normal operations
[Fri Sep 13 17:57:25.863068 2019] [core:notice] [pid 6] AH00094: Command line: 'apache2 -D FOREGROUND'

In another terminal

$ docker run -it --rm --pid=container:php tianon/network-toolbox
root@99f6f1a8470c:/# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  4.0  0.0   2388   688 ?        Ss   17:57   0:00 /bin/sh -c apache2-foreground
root         6  1.7  0.2  82732 24604 ?        S    17:57   0:00 apache2 -DFOREGROUND
www-data    17  0.0  0.0  82756  6884 ?        S    17:57   0:00 apache2 -DFOREGROUND
www-data    18  0.0  0.0  82756  6884 ?        S    17:57   0:00 apache2 -DFOREGROUND
www-data    19  0.0  0.0  82756  6884 ?        S    17:57   0:00 apache2 -DFOREGROUND
www-data    20  0.0  0.0  82756  6884 ?        S    17:57   0:00 apache2 -DFOREGROUND
www-data    21  0.0  0.0  82756  6884 ?        S    17:57   0:00 apache2 -DFOREGROUND
root        22 10.5  0.0  19068  4028 pts/0    Ss   17:57   0:00 bash --login -i
root        32  0.0  0.0  36636  2876 pts/0    R+   17:57   0:00 ps aux

[hburrows]

@wglambert Thanks a ton for the suggestions and assistance. I was already considering trying CMD to see if that would solve the problem but wasn't sure if the root shell would swallow the signal or not. I tried that suggestion and it still didn't do the trick. But it made me double back on possibly the load balancer killing that tasks and sure enough... if an application load balancer's health check fails it kills the ECS tasks. Turns out I wasn't letting port 80 outbound from the load balancer so the health check couldn't succeed. I feel like a complete idiot since I've burnt up 1-1/2 days on this (and submitted this issue)... There's been enough chatter regarding SIGWINCH recently that it sent me off on the wrong trail. Live and learn. I'm closing this issue now.