Skip to content

systemctl reload caddy.service fails with log output dial fd: unknown network fd when using socket activation #6631

New issue
New issue
Open
Open
systemctl reload caddy.service fails with log output dial fd: unknown network fd when using socket activation#6631

Description

@eriksjolund

eriksjolund
openedon Oct 15, 2024

I built git commit a211c65 (master branch as of today)

  1. Install caddy executable to /usr/local/bin/caddy
  2. Run command 
    sudo groupadd --system caddy
  3. Run command 
    sudo useradd --system \
   --gid caddy \
   --create-home \
   --home-dir /var/lib/caddy \
   --shell /usr/sbin/nologin \
   --comment "Caddy web server" \
   caddy`
  4. Create file /etc/caddy/Caddyfile containing 
    {
	auto_https disable_redirects
	admin fd/4
}

http://test.example.com {
	bind fd/3 {
		protocols h1
	}
	log
	respond "hello world
"
}
  5. Create file /etc/systemd/system/caddy.service containing 
    [Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target
    It's the same content as
    https://github.com/caddyserver/dist/blob/34f51d87d16c3da70156edffdf7c52a6916f48e6/init/caddy.service#L16-L35
    but /usr/bin/caddy replaced with /usr/local/bin/caddy
  6. Create file /etc/systemd/system/caddy.socket 
    [Socket]
ListenStream=0.0.0.0:8077
ListenStream=%t/caddy.sock
[Install]
WantedBy=sockets.target
  7. Run command 
    sudo systemctl daemon-reload
  8. Run command 
    sudo systemctl start caddy.socket`
  9. Run command 
    curl --resolve test.example.com:8077:127.0.0.1 http://test.example.com:8077
    The following output is printed 
    hello world
  10. Run command 
    sudo systemctl reload caddy.service
    The following output is printed 
    Job for caddy.service failed.
See "systemctl status caddy.service" and "journalctl -xeu caddy.service" for details.
  11. Run command 
     sudo journalctl -xeu caddy.service --no-pager | tail -79
    The following output is printed 
    Oct 15 17:26:25 fcos-next5 systemd[1]: Starting caddy.service - Caddy...
░░ Subject: A start job for unit caddy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit caddy.service has begun execution.
░░ 
░░ The job identifier is 44912.
Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.HomeDir=/var/lib/caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Oct 15 17:26:26 fcos-next5 caddy[43845]: caddy.Version=(devel)
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.GOOS=linux
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.GOARCH=arm64
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.Compiler=gc
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.NumCPU=2
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.GOMAXPROCS=2
Oct 15 17:26:26 fcos-next5 caddy[43845]: runtime.Version=go1.23.2
Oct 15 17:26:26 fcos-next5 caddy[43845]: os.Getwd=/
Oct 15 17:26:26 fcos-next5 caddy[43845]: LANG=C.UTF-8
Oct 15 17:26:26 fcos-next5 caddy[43845]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Oct 15 17:26:26 fcos-next5 caddy[43845]: NOTIFY_SOCKET=/run/systemd/notify
Oct 15 17:26:26 fcos-next5 caddy[43845]: LISTEN_PID=43845
Oct 15 17:26:26 fcos-next5 caddy[43845]: LISTEN_FDS=2
Oct 15 17:26:26 fcos-next5 caddy[43845]: LISTEN_FDNAMES=caddy.socket:caddy.socket
Oct 15 17:26:26 fcos-next5 caddy[43845]: USER=caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: LOGNAME=caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: HOME=/var/lib/caddy
Oct 15 17:26:26 fcos-next5 caddy[43845]: INVOCATION_ID=3b389f56fa5645e19bcc47c90e4df4c0
Oct 15 17:26:26 fcos-next5 caddy[43845]: JOURNAL_STREAM=9:242885
Oct 15 17:26:26 fcos-next5 caddy[43845]: SYSTEMD_EXEC_PID=43845
Oct 15 17:26:26 fcos-next5 caddy[43845]: MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/system.slice/caddy.service/memory.pressure
Oct 15 17:26:26 fcos-next5 caddy[43845]: MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA=
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1250536,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.133084,"msg":"adapted config to JSON","adapter":"caddyfile"}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1350574,"logger":"admin","msg":"admin endpoint started","address":"fd/4","enforce_origin":false,"origins":["","//127.0.0.1","//::1"]}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.135701,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1358926,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400063cb00"}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1370368,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.137156,"msg":"serving initial configuration"}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1375623,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"127.0.0.1","remote_port":"55170","client_ip":"127.0.0.1","proto":"HTTP/1.1","method":"GET","host":"test.example.com:8077","uri":"/","headers":{"User-Agent":["curl/8.9.1"],"Accept":["*/*"]}},"bytes_read":0,"user_id":"","duration":0.000027332,"size":12,"status":200,"resp_headers":{"Server":["Caddy"],"Content-Type":["text/plain; charset=utf-8"]}}
Oct 15 17:26:26 fcos-next5 systemd[1]: Started caddy.service - Caddy.
░░ Subject: A start job for unit caddy.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit caddy.service has finished successfully.
░░ 
░░ The job identifier is 44912.
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1550741,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"896930ed-628f-4cbf-8f97-981466427717","try_again":1729099586.1550682,"try_again_in":86399.999999042}
Oct 15 17:26:26 fcos-next5 caddy[43845]: {"level":"info","ts":1729013186.1552908,"logger":"tls","msg":"finished cleaning storage units"}
Oct 15 17:26:39 fcos-next5 systemd[1]: Reloading caddy.service - Caddy...
░░ Subject: A reload job for unit caddy.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A reload job for unit caddy.service has begun execution.
░░ 
░░ The job identifier is 45045.
Oct 15 17:26:39 fcos-next5 caddy[43855]: {"level":"info","ts":1729013199.412728,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Oct 15 17:26:39 fcos-next5 caddy[43855]: {"level":"info","ts":1729013199.4152887,"msg":"adapted config to JSON","adapter":"caddyfile"}
Oct 15 17:26:39 fcos-next5 caddy[43855]: Error: sending configuration to instance: performing request: Post "http://127.0.0.1/load": dial fd: unknown network fd
Oct 15 17:26:39 fcos-next5 systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecReload= process belonging to unit caddy.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Oct 15 17:26:39 fcos-next5 systemd[1]: Reload failed for caddy.service - Caddy.
░░ Subject: A reload job for unit caddy.service has finished
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A reload job for unit caddy.service has finished.
░░ 
░░ The job identifier is 45045 and the job result is failed.

Describe the results you received

The command in Step 10 failed (sudo systemctl reload caddy.service)

In the log output there is the error message

 Error: sending configuration to instance: performing request: Post "http://127.0.0.1/load": dial fd: unknown network fd

Describe the results you expected

I would have expected the command in Step 10 to succeed.

Additional information

I tried the same reproducer but with /etc/systemd/system/caddy.socket
having the following content

[Socket]
ListenStream=0.0.0.0:8077
ListenStream=%t/caddy.sock
SocketUser=caddy
SocketGroup=caddy
[Install]
WantedBy=sockets.target

The error was the same.

About the system

Fedora CoreOS 41.20241006.1.1

# rpm-ostree status
State: idle
AutomaticUpdatesDriver: Zincati
  DriverState: active; periodically polling for updates (last checked Tue 2024-10-15 17:10:11 UTC)
Deployments:
● fedora:fedora/aarch64/coreos/next
                  Version: 41.20241006.1.1 (2024-10-08T00:00:38Z)
               BaseCommit: 0dae4f8983c2c67147e4e44dcb9e0183ee8d1a301a2047fe30be33937727c7ed
             GPGSignature: Valid signature by 466CF2D8B60BC3057AA9453ED0622462E99D6AD1
      RemovedBasePackages: containerd 1.7.22-2.fc41 moby-engine 27.3.1-1.fc41 runc 2:1.1.12-4.fc41

  fedora:fedora/aarch64/coreos/next
                  Version: 40.20240906.1.0 (2024-09-10T00:50:59Z)
               BaseCommit: 0db203f6e5bdc3ec6ce57e200c84d4506e12e641c681f2212ef77e8019f1381b
             GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC
      RemovedBasePackages: containerd 1.6.23-5.fc40 moby-engine 24.0.5-4.fc40 runc 2:1.1.12-3.fc40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    bug 🐞Something isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions