Document configuration when running behind TLS-terminating Proxy

[htdvisser]

Summary

When you run The Things Stack behind a reverse proxy, you'll have to completely disable TLS in the configuration and make the proxy responsible for terminating all TLS connections (not only HTTP, but also gRPC, MQTT etc.). We should document how to disable all TLS listeners of The Things Stack, and what ports need to be mapped in the reverse proxy.

I think we can expect that people who would do this already know how their proxy works, so I don't think we should document how to do this specifically with apache/nginx/haproxy/envoy/etc.

Refs #1752 (comment)

Why do we need this ?

Because not all deployments will have The Things Stack directly exposed.

Can you do this yourself and submit a Pull Request?

Yes

[neoaggelos]

I had tried something similar a while back (for Nginx, not Apache, but it shouldn't actually matter). I think the documentation should also discuss the following points:

• How to setup the The Things Stack URLs, given that the request is being rewritten (or at least that's what it used to do? I have not checked if that has changed recently)
• How to serve The Things Stack under a URL prefix on the proxy server. This should refer to the relevant configuration options (I think --console.mount is one of them?)
• How to completely disable TLS. This probably requires a few code changes as well (I remember that The Things Stack would crash if no valid TLS certificate was given, or if some TLS port configs were empty. Not sure if this is still the case, need to check that one as well)

@htdvisser I had done some initial work on this a couple months ago. Maybe I could take up or review some parts if you are busy with other things, or if it's not very high priority. I would have to dig into multiple parts in the code that I'm not already familiar with for this.

[htdvisser]

Let's try to not broaden the scope of this issue too much. It's really easy to find more use cases that we want to cover, but then this issue will grow into something that takes a lot of time, while now it can be closed within a couple of hours (including review). Instead, we try to keep issues small, so that they can quickly be implemented and reviewed.

We can definitely create separate issues for other use cases, but let's keep this one scoped to documenting how to run a single instance of The Things Stack behind a TLS-terminating L4 proxy or load balancer.

It's also not high priority, since it's a more advanced use case. Other issues, like documenting how to connect different types of gateways and different types of end devices to The Things Stack, are much more important.

The Backlog milestone indicates that we will (at some point) work on an issue, but that we have more important things to do first (issues in the milestone for the current month, or in "Next Up").

If users want to "bump" issues, they can react with a 👍 emoji on the issue (with the +:smile: button in the top right of the issue description). That way we can sort the issues by demand, and plan accordingly.

[benolayinka]

Someone shared their configuration for nginx: #2353 (comment)

[KrishnaIyer]

Someone shared their configuration for nginx: #2353 (comment)

That's nice but it's beyond the scope of this issue. We shouldn't really care about what proxy is being used and how that's setup. We'll leave that to operators.

[nsa]

... but let's keep this one scoped to documenting how to run a single instance of The Things Stack behind a TLS-terminating L4 proxy or load balancer.

It's also not high priority, since it's a more advanced use case. Other issues, like documenting how to connect different types of gateways and different types of end devices to The Things Stack, are much more important.

I understand this may not be as important as some other issues but can you at least give a hint about when this document will be released?

[htdvisser]

This issue is currently in the "Backlog" milestone, meaning that the core team is not planning on working on it any time soon, but if anyone from the community wants to contribute such documentation, please comment here!

If you already have experience with using Apache, Nginx, Envoy or Traefik, it shouldn't be too much effort to configure the HTTP routes (/, /oauth/*, /console/*, /assets/*, /api/v3/*) and the gRPC routes (/ttn.lorawan.v3/*).

We currently don't recommend proxying the UDP or MQTT APIs through Apache, Nginx, Envoy or Traefik.

[htdvisser]

Re-assigning to @KrishnaIyer because he's been working on TLS/mTLS recently. This issue has multiple 👍 votes from users, so there is clearly demand for it.

[TomlDev]

Would be great to have this!

[aniston]

I'm still struggling with TTS v3.22.2 stack in docker that has a Traefik Proxy 2.x on the front end. I got past getting the web "console" working with letsencrypt and https but sadly a configured gateway that tries the Basics Station LNS approach with AUTH + Token cannot get past traefik (i presume logically, but cannot technically confirm)

I have a hinch it's to do with gRPC connections not getting thru, but I'm not master at TLS , sad part is I don't know where to look for errors or blocked traffic. Any hints on gRPC connection tracing to TTS v3 with traefik is valuable.

Othe rway round tonight I'll try a dry docker under debian under the same Hetzner VPS setup to make sure the TTS stack is setup correctly aith my RAK7268 gateway. I have checked in parallel that the RAK7268 is able to connect as a Basics Station to the community TTN successfully.

[KrishnaIyer]

I have a hinch it's to do with gRPC connections not getting thru,

Basic Station does not use gRPC. It uses Web Sockets (which work over an HTTP connection). Please post your issue on the forum with details.

[wereii]

From the age of this issue it seems this is not a priority but would it be possible to at least give hints here in the issue as how to completely disable tls?

[MarekNovakACRIOS]

Hello guys, I think we have a working setup with TTS running at HTTP port 8089 (non-TLS, just HTTP) with Traefik as reverse-proxy. Here is a repository with the setup. You just need to have "proxy" docker network and Traefik running on the proxy docker network. I think it could be helpful for somebody...
https://github.com/ACRIOS-Systems/ttn-docker-compose

It is also necessary to implement the healthcheck for the "stack" service, otherwise Traefik does not see the service as running, since it keeps reporting as "starting" and therefore it ignores its http server at :8089 even if the port is explicitly exposed.

I know that Docker and Traefik is just one of solutions for running TTS behind reverse proxy, but it is not a bad one, so I share it.

[KrishnaIyer]

I've added a PR to our docs to address this TheThingsIndustries/lorawan-stack-docs#1158. Please comment in the PR if there's anything else needed.