feat: Native Tailscale support for distributed coordinator & workers + networking guide

[cyklee]

Extend Dagu's built-in Tailscale tunneling capability (dagu server --tunnel) to the distributed coordinator and worker commands. Additionally, provide official documentation on the correct patterns for running Dagu's gRPC-based distributed execution over a Tailnet.

I have tried to use tailscale serve for gRPC (tcp forwarding) as well as the health ports (coordinator's 8091 and worker's 8092) without much success. Any suggestions would be greatly appreciated.

[yohamta0]

That's very interesting. Are there any specific use cases where distributed execution over Tailscale (or a Tailnet) is particularly useful?

[cyklee]

Hi @yohamta0, thanks for the quick response!

The setting I have in mind is academic research computing, where the machines that could do useful work are scattered and administered independently. In my case that means a mix of lab workstations, a couple of shared analysis servers, and the occasional cloud instance, sitting behind different departmental and institutional firewalls. Outbound SSH is reliably available, but anything that needs an inbound port, a static address, or a firewall exception means a request to central IT, and often it simply does not happen. Tailscale is my preferred answer to this, since it gives every node a stable, addressable identity on an encrypted overlay without touching any of that institutional networking, and the ACLs let me keep the coordinator and workers reachable only to each other.

The other half is observability, which is the actual reason I started looking. Right now I run pipelines across these machines with Nextflow over its SSH executor. It works, but once a task is dispatched it is effectively running in the dark, and I get very little insight into what is happening on the remote side or why something failed. Dagu's coordinator/worker model gives me the visibility I want, with task status, logs and retries surfaced centrally rather than reconstructed after the fact from a remote shell.

So the use case is roughly: execution across independently managed hosts where you have SSH but no authority over the network. Going distributed is not really a choice in my case, since the machines have different roles, a GPU box, a CPU box, a storage node, and a workstation I can easily reach to act as the coordinator. I suspect this is fairly common in university and small-lab settings, and probably anywhere people want to fold home or edge machines into a worker pool without exposing services publicly.

The baseline I really care about is coordinating work across these hosts with real visibility. SSH is the one thing I can count on having everywhere, so if that coordination could be built on top of SSH directly, that would be just as welcome. For reference, I have not managed to get tailscale serve to carry the gRPC traffic or the health ports (8091/8092) reliably, so either a native tunnel in the coordinator and worker commands or a documented SSH-based pattern would help a great deal.

[yohamta0]

Thanks for the detailed context. This use case makes a lot of sense.

I looked into the current implementation. The dagu server --tunnel currently exposes the Web UI/API HTTP server only. It does not expose the coordinator/worker gRPC path today.

For distributed execution, workers need to reach the coordinator gRPC address directly (default port is 50055). The health ports 8091 and 8092 are only HTTP health endpoints and should not be used as worker coordinator addresses.

I added a networking page with the supported patterns:
https://docs.dagu.sh/server-admin/distributed/networking

For Tailscale, I’d start with direct Tailnet connectivity. The page also covers Tailscale Serve TCP forwarding and SSH local forwarding for cases where direct connectivity is not available. Could you give it a try?

[cyklee]

Should I forward http on port 8091 on the coordinator and 8092 on the worker as well? In my previous ad hoc attempts, I ran into the issue of apparent successful TCP forwarding 50055 on the coordinator but worker would either throw errors of not getting heartbeat or, if I set health.port to 0 for worker, it would just hang. I will try following the guide and report back with proper log messages. Thanks again!

[yohamta0]

Thanks. The health ports are separate from the worker-to-coordinator connection.

8091 and 8092 are only HTTP /health endpoints (8091 for coordinator, 8092 for worker). Workers do not use them for distributed execution. The required path is:

worker -> coordinator:50055

For Tailscale Serve, I would not forward the health ports. Try this on the coordinator host:

dagu coordinator \
  --coordinator.host=127.0.0.1 \
  --coordinator.advertise=<coordinator-tailnet-name> \
  --coordinator.port=50055

tailscale serve --tcp=50055 tcp://localhost:50055

And on the worker:

dagu worker --worker.coordinators=<coordinator-tailnet-name>:50055

worker.health_port: 0 only disables the worker's own /health endpoint. If the worker then appears to sit there, it is probably still running and retrying poll/heartbeat calls to 50055.

The main thing to check is coordinator.advertise: it must be an address the worker can reach over the Tailnet.