Problem

Currently there are several approaches to .env files in the repository.

First party services

Repo root:

• only contains env.template
• .env is generated from env.template
• used by Docker Compose configs of some select services

web (API), ingestion_server:

• contains env.template and env.docker
• .env is generated from env.template
• Docker Compose uses both env.docker and .env to populate env vars

indexer_worker:

• same as ingestion server above
• except Docker Compose uses only env.docker to populate env vars

scheduler (catalog), webserver (catalog):

• only contains env.template
• .env is generated from env.template
• Docker Compose uses .env from both repo root and catalog/ to populate env vars

Docker services

db, upstream_db, plausible_db, plausible, es:

• only contains env.docker
• Docker Compose uses env.docker from docker/<folder> to populate env vars

s3, load_to_s3:

• only contains env.template
• .env is generated from env.template
• Docker Compose uses .env from both repo root and docker/minio/ to populate env vars

The effect of this is that it's very confusing to know which file to edit and how the edit will affect the service after a cascading effect of multiple env files.

Description

The solution is to use one consistent approach. Catalog's approach to this seems very good in this regard in that there is one file, env.template (checked into VCS with sensible defaults for dev), .env is generated by cloning this file (ignored from VCS) and only the .env file is used for all purposes.

If there are vars that are shared across services like AWS keys, they go in the root. Everything else stays in service-specific env.templates.

Tracking

• Running recipes in subprojects directly results in warnings about catalog-specific environment variables not existing #1832
• Clean up disparate and duplicative env.docker and root env.template #2437