Help with understanding how I can DRY my infrastructure

[hydridity]

Greetings,

I have quite the dilemma extending my Terraform workflow with Terragrunt and I'd like to ask for help with nudging me in the right direction

Context and story

We've started adopting Hashicorp Vault and with our existing declarative gitops practices the only right path was to manage it with Terraform, I dont mean deployment but purely the settings (Policies, Secret Engines, Identity Groups, Auth Methods...) .
So I've started with simple Terraform repository that managed single instance of Vault with everything inside it, that has now outgrown beyond it's initial scope, we've switched to enterprise version and started utilizing namespaces.

Entire infrastructure is divided in the concept of nested stage/customer (stage = environment: dev/test/prod...) and each stage having customers (customers are common across the stages, tho it's better to think of it in terms of region or product), and so the namespaces in the Vault do mirror it, there is namespace for each stage, and each stage namespace has sub-namespace for customer

The Dilemma

Since I'm dealing with multiple environments that has common configuration:

• Each customer in each stage has secret engines, KV engine will be always enabled, others such as transit ... depending on the need and regulations
• Each customer has OIDC login auth method enabled - varies only in the client-id and secret
• Each customer has list of applications and associated ACL policies with it - my original Terraform repository already has module that takes list via variable and generates ACL policy resource for each app and Identity Group associated with the App team

Any of those are common resources shared, varying only in the variables, so my conclusion was to reach for Terragrunt.

Until now I've been thinking about the Terragrunt repo in the scope of this only application, I've setup structure of my Terragrunt project similarly as the terragrunt-live-repo example:

.
├── _terraform
│   └── vault
│       ├── main.tf
│       ├── ...
│       └── variables.tf
├── dev
│   ├── _dev
│   │   └── terragrunt.hcl
│   ├── customer1
│   │   └── customer.hcl
│   └── stage.hcl
└── root.hcl

I'm using the _dev as alternative fot the _global structure, since the namespace for stage is standalone namespace that needs it's own configuration independent from the customer

The file stage.hcl just contains local variable stage="dev", and customer.hcl contains customer="customer1" so far.
Then in my root.hcl I'm sourcing those variables and building state name, link where to store the backend and also provider settings:

locals {
  project = "my-project"
  state-url = "[gitlab state api]/terraform/state"

  stage_vars = read_terragrunt_config(find_in_parent_folders("stage.hcl"))
  customer_vars = read_terragrunt_config(find_in_parent_folders("customer.hcl", "i-dont-exist.hcl"), {locals={"customer": local.stage}}) 
# if we run terragrunt from the _dev there is no customer.hcl in the parent folders, so we set customer to stage and we store state as "stage-stage" to preserve the pattern name

  stage = local.stage_vars.locals.stage
  customer   = local.customer_vars.locals.customer
  namespace = local.stage == local.customer ? local.stage : "${local.stage}/${local.customer}"
# if we run terragrunt from the _dev, that means we manage the /[stage] namespace, otherwise we manage /[stage]/[customer] which is separate state
}

remote_state {
    backend = "http"
    generate = {
        path = "backend.tf"
        if_exists = "overwrite_terragrunt"
    }
    config = {
        address = "${local.state-url}/${local.stage}-${local.customer}"  # i'm aware this can be single variable
        lock_address = "${local.state-url}/${local.stage}-${local.customer}/lock"
        unlock_address = "${local.state-url}/${local.stage}-${local.customer}/lock"
        lock_method = "POST"
        unlock_method = "DELETE"
        retry_wait_min = 5
    }
}

generate "provider" {
    path = "provider.tf"
    if_exists = "overwrite_terragrunt"
    contents = <<EOF
    provider "vault" {
        address = "[VAULT API URL]"
        namespace = "${local.namespace}"
    }
    EOF
}

After that I've started modularizing the original Tarraform and adding the modules into Terragrunt for environment where I needed them:

.
├── _terraform
│   └── vault_approles
│       ├── main.tf
│       ├── ...
│       └── variables.tf
├── dev
│   ├── _dev
│   │   └── terragrunt.hcl
│   ├── customer1
│   │   ├── vault_approle
│   │   │   └── terragrunt.hcl
│   │   └── customer.hcl
│   └── stage.hcl
└── root.hcl

With variable inputs and terraform source for the module in the final terragrunt.hcl.

But I've quickly realized the problem with this setup, if I add another module for different part of the configuration, each of the modules need to initialize it's own backend.
That is counterproductive(initializing the repo after clonning will be a pain) and I can imagine potential issues if all modules will use the same state since it does not make sense to split the states in this situation as most of the resources depends on each other (App Roles depend on policies, Auth engines depend on Identity groups...).
I think it's best to keep then in single state for the specific environment.

Which got me thinking, should I be considering the Terragrunt repo in more of a description of entire infrastructure stack and not just for the single application?
In a way that I'd keep the child module inside customer as whole application (vault in this case) that will have all the inputs for entire Terraform project and save it into application state, while separating the provider configuration from root.hcl to another folder per application I'd want to manage same way as the _envcommon is in the terragrunt-live-example is ?

So when I have another Terraform repository for Keycloak I could bring it as another child module for the environment and use the variables from the Terragrunt repository as common inputs for both application:

.
├── _envcommon
│   ├── keycloak.hcl # provider configuration
│   └── vault.hcl # provider configuration
├── _terraform
│   ├── keycloak
│   │   ├── main.tf
│   │   └── variables.tf
│   └── vault
│       ├── main.tf
│       └── variables.tf
├── dev
│   ├── _dev
│   │   └── terragrunt.hcl
│   ├── customer1
│   │   ├── customer.hcl
│   │   ├── keycloak
│   │   │   └── terragrunt.hcl
│   │   └── vault
│   │       └── terragrunt.hcl
│   └── stage.hcl
└── root.hcl # only common backend configuration for terraform state, keeping each application in separate state per stage

Is that a valid approach ?

[yhakbar]

Hey @hydridity ,

If it works for you and your team, it's valid!

Based on what you've written so far, it seems like you've already looked at the following:
https://terragrunt.gruntwork.io/docs/getting-started/quick-start/
https://terragrunt.gruntwork.io/docs/getting-started/overview/
https://github.com/gruntwork-io/terragrunt-infrastructure-live-example

They're the best examples for learning about the best practices Gruntwork recommends adopting when writing Terragrunt configurations.

The one major thing that I don't see you doing that most users do is use path_relative_to_include for determination of the state key in your backend. That makes it relatively easy to add new Terragrunt units, as each terragrunt.hcl file will use different state by virtue of its position in the filesystem.

Is there something about your configuration that you're particularly concerned about?