Nginix-proxy not setting up server block for LetsEncrypt ACME challenge

[davidparks21]

I'm currently trying to use nginx-proxy together with docker-letsencrypt-nginx-proxy-companion to automatically issue SSL certificates for my services. I have set up my Docker Compose configuration according to the documentation, and the containers start without any errors. However, nginx-proxy doesn't seem to be setting up the server block for the ACME challenge correctly.

When Let's Encrypt tries to validate the domain, it receives a 503 Service Unavailable error. The nginx-proxy logs show that it received the request and returned the 503 status. I suspect that this is because the server block for the ACME challenge isn't being set up correctly, but I haven't been able to figure out why this is happening.

Here's the relevant part of my Docker Compose configuration:

  nginx-proxy:
    image: jwilder/nginx-proxy:latest
    container_name: nginx-proxy
    volumes:
      - vol-certs:/etc/nginx/certs:rw  # Cache SSL certificates on the host
      - vol-vhost:/etc/nginx/vhost.d
      - vol-html:/usr/share/nginx/html
      - /var/run/docker.sock:/tmp/docker.sock:ro  # Unix socket mount to access docker API read-only
    networks:
      - braingeneers-net
    environment:
      DEBUG: "true"
      VERBOSE: "true"
      FORCE_INTERVAL: 15s
      LETSENCRYPT_HOST: braingeneers.gi.ucsc.edu
      LETSENCRYPT_EMAIL: braingeners-admins-group@ucsc.edu
      LETSENCRYPT_TEST: "false"
      DEFAULT_HOST: braingeneers.gi.ucsc.edu
      DEFAULT_PORT: "80"
    ports:
      - "80:80"
      - "443:443"

  letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: letsencrypt
    environment:
      NGINX_PROXY_CONTAINER: "nginx-proxy"
      LETSENCRYPT_TEST: "true"   # true | false for debugging | production (remember to set corresponding value in NGINX)
      DEBUG: "0"   # 1 | 0 for debugging | production (remember to set corresponding value in nginx-proxy)
      # - ACME_CA_URI=https://acme-staging-v02.api.letsencrypt.org/directory  # comment out for production use
    volumes:
      - vol-certs:/etc/nginx/certs:rw  # Shared with nginx-proxy
      - vol-vhost:/etc/nginx/vhost.d
      - vol-html:/usr/share/nginx/html
      - vol-acme.sh:/etc/acme.sh  # todo try removing this and see if it breaks, I don't think this is actually used
      - /var/run/docker.sock:/var/run/docker.sock:ro  # Unix socket mount to access docker API read-only
    depends_on:
      - nginx-proxy
    networks:
      - braingeneers-net

The services that need the SSL certificates have the VIRTUAL_HOST and LETSENCRYPT_HOST environment variables set to their respective domains. All containers are on the same Docker network.

To troubleshoot, we inspected Docker events, verified that all containers are on the same network, and checked that the environment variables are set correctly. However, we didn't observe any configuration errors or unusual events. The nginx-proxy logs only showed the 503 status for the ACME challenge requests, but no errors or warnings.

Here are some of the relevant log entries from nginx-proxy:

nginx.1     | braingeneers.gi.ucsc.edu 23.178.112.103 - - [29/Jul/2023:21:29:06 +0000] "GET /.well-known/acme-challenge/Nug808fIr6wE2ZvR1ilu0H-edOowg3Ms-1IAcgIm5rw HTTP/1.1" 503 190 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx.1     | braingeneers.gi.ucsc.edu 54.200.141.88 - - [29/Jul/2023:21:29:06 +0000] "GET /.well-known/acme-challenge/Nug808fIr6wE2ZvR1ilu0H-edOowg3Ms-1IAcgIm5rw HTTP/1.1" 503 190 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
...

Any help in figuring out why the server block for the ACME challenge isn't being set up would be greatly appreciated. I've run GPT4 out of ideas on this one. At one point I had it working by manually including a server block for the challenge files, but that change combined with a few others I ran into along the way snowballed into problems I could no longer debug, so I've backed out to the original configuration and am trying to see why things aren't working without hacking at it.

[buchdag]

Hi.

I don't think you're following proper documentations because there are no VERBOSE, FORCE_INTERVAL nor DEFAULT_PORT environment variables on the nginx-proxy container, the DEBUG environment variable has been removed a while ago and the LETSENCRYPT_* variables have no use on this container.

vol-acme.sh:/etc/acme.sh # todo try removing this and see if it breaks, I don't think this is actually used : don't remove this, this is 100% used and mandatory for data persistance, that's literally the third line of the docs 😅 .

Also it's not possible to tell if your services are correctly configured without their actual configurations.