Deprecate the env-var way of running under docker

[richvdh]

We've decided to stop supporting the configuration of synapse via environment variables.

The main reason for this is that you're bound to end up needing to do more things than it's possible to do with the environment variables, and the incomplete support leads to problems like #5430.

An alternative might have been to make it genuinely possible to configure everything via environment variables in some consistent fashion, but that sounds hard and more work than we're able to do.

[richvdh]

For clarity: we'll provide documentation on how to convert your environment-variable-configured docker image to a yaml-configured one.

[richvdh]

done by #5566.

[richvdh]

Since this is the place I point to when people say "why u no support env vars", let me say here for the record: It basically comes down to the fact that environment variables aren't powerful enough.

With the old way of configuring synapse based on a handful of env vars, people kept discovering things that weren't supported, which was annoying for them and led to support headaches for us. (Just maintaining the template for the generated config was work which we didn't really have time for, and tended to forget, since it wasn't something that we used ourselves).

Maybe you could come up with a generic scheme for mapping environment variables into configuration options, but that gets ugly quickly. For example, how would you give a (long) list of IPv6 addresses to include on the federation_ip_range_blacklist via environment variables? Or a list of HTTP listener configurations, each with several sub-options? It's possible to come up with ways of doing so, but they all feel like you're fighting a battle which shouldn't be fought.

Our experience with orchestration via ansible, kubernetes, etc is that config files make for much easier templating than magic environment variables which then get turned into configuration files via a somewhat opaque process. We recommend using your orchestration system's templating support to deploy a config file rather than somehow setting environment variables in your containers.

[jasonmp85]

This isn't about ergonomics, at least not directly. I have, I hope, a clarifying question:

In the provided docker-compose.yml there is a password, which both PostgreSQL and Synapse need to know (as server and client, respectively). I'd like to have the value of that password in one place.

Without writing a single line of executable code (I'll allow shared config files), how am I supposed to do this?

The assertion that "environment variables make things more complex" flies in the face of pretty much every piece of advice from the past decade or so. It's needing to coordinate data across multiple config files, in different formats, for different programs, in different languages that stops me in my tracks.

Environment variables are a common denominator, and even permitting us to e.g. interpolate them (as single, static values) into the yml configuration (as e.g. docker-compose.yml does) would make all of this discussion go away with what is frankly a pretty minor functional change.

[richvdh]

Environment variables are a common denominator, and even permitting us to e.g. interpolate them (as single, static values) into the yml configuration (as e.g. docker-compose.yml does) would make all of this discussion go away with what is frankly a pretty minor functional change.

Interpolation into a config file (eg a reference to ${ENV_VAR_NAME} as compose files allow) would be good (and proposed as a likely solution in #11489). However, this feature was never that: rather, it was attempting to expose every possible configuration setting via environment variables in a generic way. That's completely different.

[jessebot]

Our experience with orchestration via ansible, kubernetes, etc is that config files make for much easier templating than magic environment variables which then get turned into configuration files via a somewhat opaque process. We recommend using your orchestration system's templating support to deploy a config file rather than somehow setting environment variables in your containers.

I've been looking through some matrix Kubernetes helm charts, like typokign/matrix-chart, and a recurrent theme is plain text passwords in the values.yaml that then get stored in plain text configMaps. The standard bitnami postgresql chart supports existingSecrets for passwords, but it looks like in a bunch of these matrix helm charts that reference the bitnami postgresql chart, they seem to take the postgres password exclusively as plain text anyway, so they can then use it to create the giant config file as a k8s configMap, rather than a k8s secret, and that's probably because an entire config file as a k8s secret would need to be base64 encoded, and I honestly don't know how to do that cleanly with go templates, plus I still wouldn't know how to pull from another existing secret for the postgres password to be cleanly integrated.

Is there any way we can have just the secrets (such as postgres password, for instance) as env vars?

Then, helm charts could use env[].valueFrom.secretKeyRef in their deployment templates, and therefore they could allow existingSecret parameters in their values.yaml files, which would let us do things like just have one source of truth for the postgres password instead of supplying it once as plain text and once as a secret. I personally use the external secrets operator to then pull those k8s secrets from an external source such as 1password. This lets you open source the helm chart and encourage open source forking without requiring the use of plain text passwords. I can't allow any plain text passwords in repos, even in private repos, for clients who have to meet certain security certifications as it would come up in an audit. Helm chart forks and Argo CD templates are common files/directories we'd want to keep in revision control though.

In the meantime, for writing/improving existing helm charts, the best I can do is change the configMap to a secret, and figure out how to base64 encode the entire thing, which probably isn't the worst, but I would still need a user of the helm chart to provide passwords in plain text, because there's no way to reference an existing secret from within a secret helm template (unless there is, in which case, please let me know as that would help quite a bit). Update: I also looked into referencing a k8s secret from a configmap, but there's not been success on moving that feature forward.

I see this was talked about a bit in #14512 but that doesn't really talk about the use case in k8s specifically and that issue referenced this one anyway. I also read #5518 (comment) and #11489 but all the conversations drop off in 2022 and I can't find how people are doing this securely with k8s outside of maybe env2config, but that would still require an extra init container to do all of this.

[rltas]

Since this is the issue usually referred to when people are looking for additional environment variables, I thought I'd put my docker compose solution here: I was looking for a way to create complex configurations in our development environments, more closely resembling our production environments, by automatically merging the default generated config with a more complex template. This is the approach I ended up with (simplified just to give the idea):

Generate Config

compose.yaml:

services:
  generate:
    build:
      context: .
      dockerfile: docker/synapse/Dockerfile.generate
    # Only run this service when explicitly requested
    profiles:
      - generate
    environment:
      SYNAPSE_SERVER_NAME: ${SYNAPSE_SERVER_NAME:?err}
      SYNAPSE_REPORT_STATS: no
      SYNAPSE_DB_PASSWORD: ${SYNAPSE_DB_PASSWORD:?err}
    volumes:
      - ./data/synapse:/data

The variables obviously have to be set via .env or the environment loaded via env_file. Also I didn't care about secrets in a strictly local WSL based development environment.

Dockerfile:

FROM matrixdotorg/synapse:latest

RUN curl -fsSL "https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64" -o /usr/bin/yq && chmod +x /usr/bin/yq

COPY ./docker/synapse/homeserver.yaml.tmpl /conf/
COPY ./docker/synapse/generate.sh /
RUN chmod +x /generate.sh

ENTRYPOINT ["/generate.sh"]

generate.sh:

#!/bin/bash

# Generate basic config (what the original entrypoint did)
/start.py generate

cp /conf/homeserver.yaml.tmpl /data/

# Strip all comments and blank lines from input files
sed -i -E '/^\s*$|^\s*#/d' /data/homeserver.yaml
sed -i -E '/^\s*$|^\s*#/d' /data/homeserver.yaml.tmpl

# Delete conflicting keys and arrays from basic config
yq -i 'del(.listeners, .database, .trusted_key_servers)' /data/homeserver.yaml

# Merge and sort the input files alphabetically
yq -i '. *= load("/data/homeserver.yaml") | sort_keys(..)' /data/homeserver.yaml.tmpl

# Replace all variables using environment variables
yq -i '(.. | select(tag == "!!str")) |= envsubst' /data/homeserver.yaml.tmpl

mv /data/homeserver.yaml.tmpl /data/homeserver.yaml

homeserver.yaml.tmpl:

database:
  args:
    cp_max: 10
    cp_min: 5
    database: synapse
    host: synapse-db
    password: "${SYNAPSE_DB_PASSWORD}"
    port: 5432
    user: synapse
  name: psycopg2
  txn_limit: 0

This is obviously an extremely reduced sample.

That approach works very well for our case.
https://github.com/mikefarah/yq is a fantastic tool.