add docs

ffddorf · Mar 21, 2022 · 565ec08 · 565ec08
1 parent 07b358c
commit 565ec08
Show file tree

Hide file tree

Showing 6 changed files with 299 additions and 0 deletions.
diff --git a/README.md b/README.md
@@ -0,0 +1,69 @@
+# Terraform Backend Server
+
+A state backend server which implements the Terraform HTTP backend API with plugable modules for authentication, storage, locking and state encryption.
+
+Supported authentication methods:
+- HTTP basic auth
+- JSON Web Tokens
+
+Supported storage backends:
+- local file system
+- S3
+
+Supported lock backends:
+- local map
+- Redis
+
+Supported KMS (encryption) backends:
+- local AES key
+- AES from HashiCorp Vault Key/Value store (v2)
+- HashiCorp Vault Transit engine
+
+**Disclaimer**
+This code is in an early development state and not tested extensively for bugs and security issues. If you find some, please raise an issue or merge request.
+
+## Deployment
+
+Run locally for development:
+```sh
+LOG_LEVEL=debug go run cmd/terraform-backend.go
+```
+
+or use [docker-compose](./docker-compose.yml):
+```sh
+docker-compose up -d
+```
+
+### Default settings
+
+The following table describes the default configuration, although the backend server will run with these values, it's not scalable and therefore only for testing purposes. 
+
+| Environment Variable | Type   | Default    | Description                                                                                       |
+|----------------------|--------|------------|---------------------------------------------------------------------------------------------------|
+| LOG_LEVEL            | string | `info`     | Log level (options are: `fatal`, `info`, `warning`, `debug`, `trace`)                             |
+| LISTEN_ADDR          | string | `:8080`    | Address the HTTP server listens on                                                                |
+| STORAGE_BACKEND      | string | `fs`       | Module for state file storage (checkout [docs/storage.md](./docs/storage.md) for other options)   |
+| STORAGE_FS_DIR       | string | `./states` | File system directory for `fs` storage module to store state files                                |
+| KMS_BACKEND          | string | `local`    | Module used for encryption (checkout [docs/kms.md](./docs/kms.md) for other options)              |
+| KMS_KEY              | string | --         | Key for `local` KMS module, if not defined, the server will generate a new one and exit           |
+| LOCK_BACKEND         | string | `local`    | Module used for locking the state (checkout [docs/lock.md](./docs/lock.md) for other options)     |
+| AUTH_BASIC_ENABLED   | bool   | `true`     | HTTP basic auth is enabled by default (checkout [docs/auth.md](./docs/auth.md) for other options) |
+
+## Usage
+
+The path to the state is: `/state/<project-id>/<state-name>`.
+
+**Example Terraform backend configuration**
+```hcl
+terraform {
+  backend "http" {
+    address        = "http://localhost:8080/state/project1/example"
+    lock_address   = "http://localhost:8080/state/project1/example"
+    unlock_address = "http://localhost:8080/state/project1/example"
+    username       = "basic"
+    password       = "some-random-secret"
+  }
+}
+```
+
+For more information about username and password checkout [docs/auth.md](./docs/auth.md)
diff --git a/docker-compose.yml b/docker-compose.yml
@@ -0,0 +1,25 @@
+version: "3"
+
+services:
+  terraform_backend:
+    image: ghcr.io/nimbolus/terraform-backend
+    build: .
+    environment:
+      STORAGE_FS_DIR: /states
+      # change the key before using the backend
+      KMS_KEY: jwS6UpASMOWpEmFn7C6I47BlmPt4cpdmYLKd2E7a4Zk=
+      REDIS_ADDR: redis:6379
+      LOCK_BACKEND: redis
+    ports:
+      - "8080:8080"
+    volumes:
+      - states:/states
+    links:
+      - redis
+  redis:
+    image: redis:alpine
+    ports:
+      - "6379:6379"
+
+volumes:
+  states:
diff --git a/docs/auth.md b/docs/auth.md
@@ -0,0 +1,109 @@
+# Authentication Backends
+
+The authentication method is defined by the HTTP basic auth username, therefore multiple authentication method can be used to protect different Terraform states.
+
+## HTTP Basic Auth
+
+This authentication creates a hash value of provided HTTP basic auth password and state path to get the filename of the state. Therefore only the right combination of state path and password can fetch this exact state again. It's really simple to setup, no user or credential management required. The drawback is that the server can be used by everyone, who has access to the API endpoint, so it should only be used in secure or testing environments.
+
+**Config**
+| Environment Variable | Type | Example | Description                                                                                     |
+|----------------------|------|---------|-------------------------------------------------------------------------------------------------|
+| AUTH_BASIC_ENABLED   | bool | `true`  | HTTP basic auth is enabled by default (checkout [docs/auth.md](docs/auth.md) for other options) |
+
+**Example Terraform backend configuration**
+```hcl
+terraform {
+  backend "http" {
+    address        = "https://<terraform-state-server>/state/project1/example"
+    lock_address   = "https://<terraform-state-server>/state/project1/example"
+    unlock_address = "https://<terraform-state-server>/state/project1/example"
+    username       = "basic"
+    password       = "some-random-secret"
+  }
+}
+```
+
+## JSON Web Tokens
+
+JWT allow granting access to a state for a given time (the token lifetime). The project and ID of the state must be part of the `terraform-backend` token claim.
+
+`terraform-backend` token claim format:
+```json
+{
+    "terraform-backend": {
+        "project": "project1",
+        "state": "example"
+    }
+}
+```
+
+**Config**
+| Environment Variable     | Type | Example                                      | Description                                                                       |
+|--------------------------|------|----------------------------------------------|-----------------------------------------------------------------------------------|
+| AUTH_JWT_OIDC_ISSUER_URL | bool | `https://vault.example.com/v1/identity/oidc` | Issuer URL which is used to validate token (if not defined, JWT auth is disabled) |
+
+
+**Example Terraform backend configuration**
+```hcl
+terraform {
+  backend "http" {
+    address        = "https://<terraform-state-server>/state/project1/example"
+    lock_address   = "https://<terraform-state-server>/state/project1/example"
+    unlock_address = "https://<terraform-state-server>/state/project1/example"
+    username       = "jwt"
+    password       = "<json-web-token>"
+  }
+}
+```
+
+### Example using HashiCorp Vault Identity Tokens
+
+HashiCorp Vault allows creating [Identity Tokens](https://www.vaultproject.io/docs/secrets/identity/identity-token) for third party systems, so that Vault policies can be used to give access to specific Terraform states.
+
+Terraform code for creating a OIDC role for accessing the state at `https://<terraform-state-server>/state/project1/exmaple`:
+```hcl
+resource "vault_identity_oidc_key" "example" {
+  name      = "example"
+  algorithm = "RS256"
+}
+
+resource "vault_identity_oidc_role" "example" {
+  name = "example"
+  key  = vault_identity_oidc_key.example.name
+  # token is valid for one hour
+  ttl  = 3600
+
+  template = <<-EOT
+    {
+      "terraform-backend": {
+        "project": "project1",
+        "state": "example"
+      }
+    }
+    EOT
+}
+
+resource "vault_identity_oidc_key_allowed_client_id" "example" {
+  key_name          = vault_identity_oidc_key.example.name
+  allowed_client_id = vault_identity_oidc_role.example.client_id
+}
+```
+
+Terraform backend configuration
+```hcl
+terraform {
+  backend "http" {
+    address        = "https://<terraform-state-server>/state/project1/example"
+    lock_address   = "https://<terraform-state-server>/state/project1/example"
+    unlock_address = "https://<terraform-state-server>/state/project1/example"
+    username       = "jwt"
+  }
+}
+```
+
+Terraform backend initialization (requires initialized HashiCorp Vault CLI client session):
+```sh
+TOKEN=$(vault read -field token identity/oidc/token/example)
+terraform init -backend-config="password=$TOKEN" -reconfigure
+```
diff --git a/docs/kms.md b/docs/kms.md
@@ -0,0 +1,45 @@
+# KMS Backends
+
+The KMS backend is responsible for encrypting and decrypting the state file, so that the stored data is always encrypted at rest.
+
+## Local Key
+
+This backend uses a key defined as an environment variable to encrypt the state files.
+
+A key can be generated by running: `./terraform-backend`
+
+**Config**
+Set `KMS_BACKEND` to `local`.
+
+| Environment Variable | Type   | Example                                        | Description                                                                                 |
+|----------------------|--------|------------------------------------------------|---------------------------------------------------------------------------------------------|
+| KMS_KEY              | string | `jwS6UpASMOWpEmFn7C6I47BlmPt4cpdmYLKd2E7a4Zk=` | If local KMS is enabled, but no key is defined, the server will generate a new one and exit |
+
+## Key from Vault Key/Value Secrets Engine
+
+Alternatively the key can be fetched from a [HashiCorp Vault Key/Value secrets engine (v2)](https://www.vaultproject.io/docs/secrets/kv/kv-v2)
+
+**Config**
+Set `KMS_BACKEND` to `vault`.
+
+| Environment Variable | Type   | Example                     | Description            |
+|----------------------|--------|-----------------------------|------------------------|
+| KMS_VAULT_KEY_PATH   | string | `kv/data/terraform-backend` | Path of the key secret |
+
+Make sure that `VAULT_ADDR` and `VAULT_TOKEN` are set properly (see [Vault Environment Variables](https://www.vaultproject.io/docs/commands#environment-variables) for more information).
+
+## Vault Transit Secrets Engine
+
+[HashiCorp Vault Transit secrets engine](https://www.vaultproject.io/docs/secrets/transit) allows delegating the en-/decryption process to a Vault server, so that even the Terraform backend server doesn't know the key.
+
+For preparing the disaster recovery, the [Transit key can be exported](https://www.vaultproject.io/api-docs/secret/transit#export-key) and the state files can be converted to use a local key for decryption by using the [convert-transit-state.sh](../scripts/convert-transit-state.sh) script.
+
+**Config**
+Set `KMS_BACKEND` to `transit`.
+
+| Environment Variable | Type   | Example   | Description                                      |
+|----------------------|--------|-----------|--------------------------------------------------|
+| KMS_TRANSIT_ENGINE   | string | `transit` | Name (mount point) of the Transit secrets engine |
+| KMS_TRANSIT_KEY      | string | `transit` | Name of the Transit key                          |
+
+Make sure that `VAULT_ADDR` and `VAULT_TOKEN` are set properly (see [Vault Environment Variables](https://www.vaultproject.io/docs/commands#environment-variables) for more information).
diff --git a/docs/lock.md b/docs/lock.md
@@ -0,0 +1,19 @@
+# Lock Backends
+
+The lock backend takes care of locking a specific state file, so that only one Terraform entity can access and change it in a given time.
+
+## Local Map
+
+This is the simplest implementation by using a local Golang map and doesn't require any configuration. It works fine for a standalone, single-instance Terraform backend server, but doesn't scale. Also if the Terraform backend server crashes, the lock information will be lost.
+
+**Config**
+Set `LOCK_BACKEND` to `local`.
+
+## Redis
+
+This backend uses a external Redis server to lock the states. It's scalable and can be used also with multiple Terraform backend server instances.
+
+**Config**
+Set `LOCK_BACKEND` to `redis`.
+
+Make sure that `REDIS_ADDR` is set properly (e.g. to `localhost:6379` for a local Redis instance).
diff --git a/docs/storage.md b/docs/storage.md
@@ -0,0 +1,32 @@
+## Storage backends
+
+The storage backend stores the state locally or remotely (depending on the implementation).
+
+NOTE: The state path is always hashed, so getting the state name of project from the file or object name isn't possible.
+
+## Local File System
+
+This backend saves the state file to a local directory.
+
+**Config**
+Set `STORAGE_BACKEND` to `fs`.
+
+| Environment Variable | Type   | Default    | Description                          |
+|----------------------|--------|------------|--------------------------------------|
+| STORAGE_FS_DIR       | string | `./states` | Local directory to store state files |
+
+## S3 Object Storage
+
+The S3 backend stores the state files in any S3-compatible object store using the [MinIO SDK](https://docs.min.io/docs/golang-client-quickstart-guide.html). Since locking is handled by the Terraform backend server separately, the S3 API doesn't need support for write-once-read-many (WORM).
+
+**Config**
+Set `STORAGE_BACKEND` to `fs`.
+
+| Environment Variable  | Type   | Default            | Description             |
+|-----------------------|--------|--------------------|-------------------------|
+| STORAGE_S3_ENDPOINT   | string | `s3.amazonaws.com` | S3 endpoint             |
+| STORAGE_S3_USE_SSL    | string | `true`             | Use SSL for S3 endpoint |
+| STORAGE_S3_ACCESS_KEY | string | --                 | S3 Access key ID        |
+| STORAGE_S3_SECRET_KEY | string | --                 | S3 Secret key           |
+| STORAGE_S3_BUCKET     | string | `terraform-state`  | Name of the S3 bucket   |
+