Configuration Guide

This project uses a multi-layered environment configuration system to separate shell interpolation, container environment variables, and deployment-specific secrets.

Table of Contents

• Two Configuration Patterns
• Shell Interpolation vs Container Environment
• The .local Override Pattern
• Environment Variable Precedence
• Best Practices
• Troubleshooting

Two Configuration Patterns

The project uses two different patterns for environment configuration:

1. Root-Level .env Files (Shell Interpolation)

Location: .env in each service directory (e.g., /srv/compose/database/.env)

Purpose: Variables used for shell interpolation in compose.yaml files

When it's read: By Docker Compose before parsing the YAML file

Example:

# compose.yaml
services:
    mongodb:
        image: mongodb/mongodb-community-server:${MONGO_TAG}
        volumes:
            - ${DATA_DIR}/mongo:/data/db # ${DATA_DIR} replaced during YAML parsing
        user: "${UID}:${GID}"

Usage:

• Image tags: ${MONGO_TAG}, ${PGVECTOR_TAG}
• Volume paths: ${DATA_DIR}
• User/group IDs: ${UID}, ${GID}
• Port mappings: ${EXTERNAL_PORT}
• Network names: ${NETWORK_PREFIX}

Important: These variables are NOT passed into containers unless explicitly defined in environment: or env_file: sections.

2. env/*.env Files (Container Environment)

Location: env/service.env files in subdirectories (e.g., /srv/compose/database/env/mongodb.env)

Purpose: Variables passed into the container as environment variables

When it's read: After compose.yaml is parsed, loaded into the container at runtime

Example:

# compose.yaml
services:
    mongodb:
        env_file:
            - ./env/mongodb.env # Base configuration
            - ./env/mongodb.env.local # Secrets and overrides

# env/mongodb.env
MONGO_INITDB_ROOT_USERNAME=admin
MONGO_INITDB_DATABASE=default

# env/mongodb.env.local (gitignored)
MONGO_INITDB_ROOT_PASSWORD=super_secret_password

Usage:

• Database credentials
• Application configuration
• API keys and tokens
• Feature flags
• Debug settings

Shell Interpolation vs Container Environment

Understanding the difference is critical for proper configuration:

Example Compose File

services:
    pgvector:
        # Shell interpolation (from .env)
        image: pgvector/pgvector:${PGVECTOR_TAG}

        # Container environment (from env/*.env)
        env_file:
            - ./env/pgvector.env
            - ./env/pgvector.env.local

        # Both! Interpolated THEN set in container
        environment:
            POSTGRES_DB: ${POSTGRES_DB}

        # Shell interpolation only
        volumes:
            - ${DATA_DIR}/postgres:/var/lib/postgresql

        # Shell interpolation only
        user: "${UID}:${GID}"

Shell Interpolation (from .env)

Happens: Before Docker Compose processes the YAML Used for: Compose file structure Variables: NOT visible inside containers (unless explicitly passed)

Common uses:

• ${DATA_DIR} - Data storage path
• ${UID}, ${GID} - User/group IDs
• ${GPU_ID} - GPU device selection
• ${NETWORK} - Network names
• ${TAG} - Image tags

Container Environment (from env_file:)

Happens: At container runtime Used by: Application running inside the container Variables: Visible inside container via env

Common uses:

• POSTGRES_DB, POSTGRES_USER - Database config
• MONGO_MAX_POOL_SIZE - Application settings
• QDRANT__SERVICE__HTTP_PORT - Service ports
• API_KEY, SECRET_TOKEN - Credentials

The .local Override Pattern

The project uses .local files for machine-specific overrides and secrets.

Base Configuration (Committed to Git)

# .env (root level - shell interpolation)
DATA_DIR=/srv/appdata
UID=1000
GID=1000
MONGO_TAG=latest

# env/mongodb.env (container environment)
MONGO_INITDB_ROOT_USERNAME=admin
MONGO_INITDB_DATABASE=myapp

Local Overrides (NOT Committed - Gitignored)

# .env.local (optional - shell interpolation overrides)
DATA_DIR=/mnt/storage/appdata
GPU_ID=1

# env/mongodb.env.local (container environment - SECRETS!)
MONGO_INITDB_ROOT_PASSWORD=my_super_secret_password

How Docker Compose Merges Them

Shell Interpolation (.env files):

1. Reads .env first
2. Reads .env.local second (overrides .env values)
3. Interpolates variables into compose.yaml

Container Environment (env_file:):

env_file:
    - ./env/service.env # Base configuration
    - ./env/service.env.local # Overrides and secrets

• Files listed later override earlier values
• Both files loaded into container
• Last value wins for duplicate keys

Why This Pattern?

✅ Separation of Secrets: Base config in git, secrets in .local files (gitignored)

✅ Machine-Specific Settings: Different paths, GPU IDs, domains per deployment

✅ Team Collaboration: Share base config, customize locally

✅ Security: Never commit passwords, API keys, or tokens

✅ Flexibility: Same compose files work across environments

Environment Variable Precedence

Docker Compose loads environment variables in this order (last wins):

1. .env file (root level - shell interpolation)
2. .env.local file (root level overrides)
3. env_file: entries in compose.yaml (container environment)
4. environment: section in compose.yaml (container environment)
5. Command-line -e flags (highest priority)

Example

# .env
DATA_DIR=/srv/appdata
APP_DEBUG=false

# .env.local
DATA_DIR=/mnt/storage  # ← Wins for shell interpolation

# env/service.env
APP_DEBUG=false
LOG_LEVEL=info

# env/service.env.local
APP_DEBUG=true  # ← Wins for container environment
LOG_LEVEL=debug # ← Wins

# compose.yaml environment section
environment:
  APP_MODE: production  # ← Always wins unless -e flag used

# Command line (highest priority)
docker compose -e APP_DEBUG=false up  # ← Overrides everything

Best Practices

DO ✅

1. Use .env for structure: Image tags, paths, UIDs

   DATA_DIR=/mnt/data
   UID=1000
   PGVECTOR_TAG=16

2. Use env/*.env for application config: Database settings, app config

   # env/postgres.env
   POSTGRES_DB=myapp
   POSTGRES_USER=appuser

3. Use .local files for secrets: Passwords, API keys, tokens

   # env/postgres.env.local
   POSTGRES_PASSWORD=secret123

4. Commit base files: .env and env/*.env

   git add .env env/*.env
   git commit -m "Add base configuration"

5. Gitignore secrets: Ensure .local files are ignored

   # .gitignore
   *.local
   *.secret
   secret/

6. Document required variables: Comment in base files

   # .env
   # Override in .env.local for custom data directory
   DATA_DIR=/srv/appdata

DON'T ❌

1. Don't commit secrets: Never add .local files to git

   # BAD!
   git add .env.local  # Contains passwords!

2. Don't mix purposes: Keep shell vars in .env, container vars in env/

   # BAD - mixing concerns
   # .env
   DATA_DIR=/srv/data
   POSTGRES_PASSWORD=secret  # Should be in env/*.env.local!

3. Don't hardcode secrets: Use environment variables

   # BAD!
   environment:
       API_KEY: "hardcoded-secret-key"
   
   # GOOD!
   env_file:
       - ./env/service.env.local

4. Don't duplicate variables: Use inheritance

   # GOOD
   # .env
   BASE_PATH=/srv/compose
   
   # BAD - duplicating BASE_PATH everywhere
   PANEL_PATH=/srv/compose/panel
   DATABASE_PATH=/srv/compose/database

Configuration Workflow

Initial Setup

# 1. Clone repository
git clone https://github.com/user/compose.git /srv/compose

# 2. Copy base configuration
cd /srv/compose/database
cp .env .env.local

# 3. Edit for your environment
nano .env.local
# Set: DATA_DIR, UID, GID, etc.

# 4. Create secrets
cp env/mongodb.env env/mongodb.env.local
nano env/mongodb.env.local
# Set: MONGO_INITDB_ROOT_PASSWORD

# 5. Verify gitignore
git status
# Should NOT show *.local files

Updating Configuration

# Update base configuration (committed)
nano .env
git add .env
git commit -m "Update default data directory"

# Update local overrides (not committed)
nano .env.local
# Changes stay local

# Update application config
nano env/mongodb.env
git add env/mongodb.env
git commit -m "Change default database name"

# Update secrets (not committed)
nano env/mongodb.env.local
# Secrets stay local

Troubleshooting

Problem: "Variable not set" in compose.yaml

Error:

ERROR: The Compose file is invalid because:
Invalid interpolation format for "volumes": "${DATA_DIR}/mongo:/data/db"
DATA_DIR is not set

Solution: Add to .env or .env.local (shell interpolation)

echo "DATA_DIR=/mnt/data" >> .env.local

Problem: Application complains about missing environment variable

Error (inside container):

Error: POSTGRES_PASSWORD environment variable is required

Solution: Add to env/*.env.local (container environment)

echo "POSTGRES_PASSWORD=mypassword" >> env/pgvector.env.local

Problem: Variable not overriding

Scenario: Set DATA_DIR in .env.local but still using .env value

Solutions:

1. Check file is named correctly: Must be exactly .env.local
2. Check file location: Must be in same directory as compose.yaml
3. Restart compose: Changes require restart

   compose down
   compose up -d

Problem: Secrets committed to git

Prevention:

# Verify gitignore
cat .gitignore | grep -E '\.local|\.secret'

# Check what's tracked
git ls-files | grep -E '\.local|\.secret'

# If secrets were committed, remove from history
git filter-branch --force --index-filter \
  'git rm --cached --ignore-unmatch env/*.local' HEAD

Debugging Environment Variables

Check shell interpolation (before compose parses):

# Load .env files manually
export $(cat .env | xargs)
export $(cat .env.local | xargs)

# Check what compose sees
compose config | grep -A5 volumes

Check container environment (inside running container):

# List all environment variables
docker exec container-name env

# Check specific variable
docker exec container-name printenv POSTGRES_PASSWORD

Check which files are loaded:

# See compose file interpretation
compose config

# See environment file contents (careful with secrets!)
cat env/service.env

Directory Structure Example

/srv/compose/database/
├── compose.yaml              # Service definitions
├── .env                      # Shell vars (committed)
├── .env.local                # Shell overrides (gitignored)
├── env/
│   ├── mongodb.env           # MongoDB config (committed)
│   ├── mongodb.env.local     # MongoDB secrets (gitignored)
│   ├── pgvector.env          # Postgres config (committed)
│   ├── pgvector.env.local    # Postgres secrets (gitignored)
│   ├── qdrant.env            # Qdrant config (committed)
│   └── qdrant.env.local      # Qdrant secrets (gitignored)
└── data/                     # Persistent data (gitignored)

Example Configuration

Panel Service

# /srv/compose/panel/.env (committed)
ACME_DOMAIN=example.com
TRAEFIK_TAG=v3.0
PORTAINER_TAG=latest

# /srv/compose/panel/.env.local (gitignored)
ACME_EMAIL=admin@example.com
CF_DNS_SECRET_FILE=./secret/cf_dns.secret

Database Service

# /srv/compose/database/.env (committed)
DATA_DIR=/srv/appdata
UID=1000
GID=1000
MONGO_TAG=7
PGVECTOR_TAG=16
QDRANT_TAG=latest

# /srv/compose/database/.env.local (gitignored)
DATA_DIR=/mnt/storage/compose
GPU_ID=0

# /srv/compose/database/env/mongodb.env (committed)
MONGO_INITDB_ROOT_USERNAME=admin
MONGO_INITDB_DATABASE=default

# /srv/compose/database/env/mongodb.env.local (gitignored)
MONGO_INITDB_ROOT_PASSWORD=super_secret_mongo_password

# /srv/compose/database/env/pgvector.env (committed)
POSTGRES_DB=postgres
POSTGRES_USER=postgres

# /srv/compose/database/env/pgvector.env.local (gitignored)
POSTGRES_PASSWORD=super_secret_postgres_password

---

Related Pages:

• Quick Start - Initial setup
• Environment Variables - Complete variable reference
• Troubleshooting - Common configuration issues