after-terraform-apply hook corrupts backend.tf.json when backend uses !terraform.state

[zack-is-cool]

Describe the Bug

Starting in Atmos v1.216.0, after-terraform-apply hooks (e.g. store hooks that write Terraform outputs to SSM) fail for components whose backend.tf.json is generated from another component's outputs via the !terraform.state YAML function.

On the initial atmos terraform apply:

1. Atmos correctly renders backend.tf.json — the !terraform.state calls resolve to real values, terraform init succeeds, terraform apply succeeds.
2. Atmos then fires the after-terraform-apply hook (a store hook that needs to read outputs from the just-applied component).
3. During the hook, Atmos regenerates backend.tf.json, but this time the YAML functions are not rendered — the literal !terraform.state ... string is written into the backend config.
4. Because the backend file is now corrupt, Atmos/Terraform thinks it needs to re-init, and the hook fails to read outputs.

Example of the corrupt backend.tf.json written during the hook:

{
  "terraform": {
    "backend": {
      "s3": {
        "bucket": "!terraform.state tfstate-backend atmos-dev s3_bucket_id",
        "dynamodb_table": "!terraform.state tfstate-backend atmos-dev dynamodb_table_name",
        "encrypt": true,
        "key": "terraform.tfstate",
        "profile": "",
        "region": "us-gov-west-1",
        "workspace_key_prefix": "app-tst"
      }
    }
  }
}

This regression was introduced by #2309, which fixed "Bug 3 – hooks fired on every event regardless of events: list." Prior to that fix, hooks fired on every event, which apparently masked this race/ordering issue (the backend was being regenerated enough times, or in a different order, that the rendered version won out). With #2309 in place and hooks correctly scoped to after-terraform-apply, the regeneration path inside the hook produces an un-rendered backend file.

This looks like either:

• a race condition between the hook's backend regeneration and YAML function evaluation, or
• the hook's regeneration code path skipping/not invoking YAML function rendering (so !terraform.state is written as a literal string), or
• the hook shouldn't be regenerating backend.tf.json at all if it already exists and is valid.

Expected Behavior

after-terraform-apply hooks should succeed for components whose backend is generated by !terraform.state (or other YAML functions). Specifically:

• The backend file produced during the hook's execution should be identical to the one produced during plan/apply — i.e. YAML functions fully rendered.
• Ideally, the hook should not regenerate backend.tf.json at all unless it has changed; at minimum, when it does regenerate, YAML functions must be resolved exactly as they are for plan/apply.

Steps to Reproduce

1. Define two components:

   • tfstate-backend: outputs s3_bucket_id and dynamodb_table_name.
   • my-app (e.g. an RDS component): its stack config generates backend.tf.json using !terraform.state to look up tfstate-backend's outputs, and has an after-terraform-apply store hook.

   Example stack config for my-app:

   components:
     terraform:
       my-app:
         backend_type: s3
         backend:
           s3:
             bucket: !terraform.state tfstate-backend atmos-dev s3_bucket_id
             dynamodb_table: !terraform.state tfstate-backend atmos-dev dynamodb_table_name
             encrypt: true
             key: terraform.tfstate
             region: us-gov-west-1
             workspace_key_prefix: app-tst
         hooks:
           store-outputs:
             events:
               - after-terraform-apply
             command: store
             outputs:
               rds_properties: .rds_properties

2. Apply tfstate-backend first so its state exists.

3. Run atmos terraform apply my-app -s <stack> on Atmos >= v1.216.0.

4. Observe: terraform apply succeeds, but the after-terraform-apply hook fails. Inspect the component's working dir and observe that backend.tf.json now contains literal !terraform.state ... strings.

Screenshots

Captured from a self-contained repro (see Additional Context below for the script). backend.tf.json is rendered correctly by atmos terraform init, then the after-terraform-apply hook overwrites it with unresolved YAML-function strings:

Before hook (correctly rendered by init):

{
  "terraform": {
    "backend": {
      "s3": {
        "bucket": "atmos-tfstate-dev",
        "dynamodb_table": "atmos-tfstate-lock-dev",
        "encrypt": true,
        "key": "terraform.tfstate",
        "region": "us-east-1",
        "workspace_key_prefix": "app-tst",
        "...": "..."
      }
    }
  }
}

After after-terraform-apply hook fires (corrupted):

{
  "terraform": {
    "backend": {
      "s3": {
        "bucket": "!terraform.state tfstate-backend dev s3_bucket_id",
        "dynamodb_table": "!terraform.state tfstate-backend dev dynamodb_table_name",
        "encrypt": true,
        "key": "terraform.tfstate",
        "region": "us-east-1",
        "workspace_key_prefix": "app-tst",
        "...": "..."
      }
    }
  }
}

Diff:

-        "bucket": "atmos-tfstate-dev",
-        "dynamodb_table": "atmos-tfstate-lock-dev",
+        "bucket": "!terraform.state tfstate-backend dev s3_bucket_id",
+        "dynamodb_table": "!terraform.state tfstate-backend dev dynamodb_table_name",

Observed hook error (exact output from atmos in the repro):

Running hooks event=after.terraform.apply
Fetching rds_properties output from my-app in dev
✗ Fetching rds_properties output from my-app in dev
Error: failed to retrieve terraform outputs: hook "local/redis" (event "after.terraform.apply")
failed to get terraform output "rds_properties" for component "my-app" in stack "dev":
failed to retrieve terraform outputs: exit status 1
  Error: Backend initialization required: please run "tofu init"
  Reason: Backend configuration block has changed

Environment

• Atmos version: >= v1.216.0 (regression introduced by fix: respect workdir path for generate: writes and hook-triggered terraform #2309)
• Last known good: < v1.216.0 (hooks fired on every event, which masked the issue)
• Terraform backend: S3 (with DynamoDB lock table)
• OS: Linux (also reproducible on macOS)

Additional Context

• Introduced by fix: respect workdir path for generate: writes and hook-triggered terraform #2309 (fixed "Bug 3 – hooks fired on every event regardless of events: list").
• Only affects components that use YAML functions (!terraform.state, and likely !terraform.output, !store, etc.) to generate their backend.tf.json (and potentially varfiles as well — needs verification).
• Hook types affected: at minimum after-terraform-apply store hooks. Other after-* hooks may be similarly affected.
• Suspected root cause: the code path that regenerates backend.tf.json prior to hook execution bypasses YAML-function rendering, or re-reads the stack config without re-rendering templates/YAML functions. A fix may involve either (a) skipping regeneration if a valid backend already exists on disk, or (b) ensuring the regeneration path reuses the same rendered config that plan/apply used.
• Varfile regeneration during the hook should also be checked — if varfiles are similarly regenerated with unresolved YAML functions, the failure mode will be broader than just the backend.