Introduce Atmos YAML functions

[aknysh]

what

• Introduce Atmos YAML functions
• Update docs
  • https://pr-810.atmos-docs.ue2.dev.plat.cloudposse.org/core-concepts/stacks/yaml-functions/
  • https://pr-810.atmos-docs.ue2.dev.plat.cloudposse.org/core-concepts/stacks/yaml-functions/template/
  • https://pr-810.atmos-docs.ue2.dev.plat.cloudposse.org/core-concepts/stacks/yaml-functions/exec/
  • https://pr-810.atmos-docs.ue2.dev.plat.cloudposse.org/core-concepts/stacks/yaml-functions/terraform.output/

why

Atmos YAML Functions are a crucial part of Atmos stack manifests. They allow you to manipulate data and perform operations on the data to customize the stack configurations.

Atmos YAML functions are based on YAML Explicit typing and user-defined Explicit Tags (local data types). Explicit tags are denoted by the exclamation point ("!") symbol. Atmos detects the tags in the stack manifests and executes the corresponding functions.

NOTE: YAML data types can be divided into three categories: core, defined, and user-defined. Core are ones expected to exist in any parser (e.g. floats, ints, strings, lists, maps). Many more advanced data types, such as binary data, are defined in the YAML specification but not supported in all implementations. Finally, YAML defines a way to extend the data type definitions locally to accommodate user-defined classes, structures, primitives, and functions.

Atmos YAML functions

• The !template YAML function can be used to handle template outputs containing maps or lists returned from the atmos.Component template function

• The !exec YAML function is used to execute shell scripts and assign the results to the sections in Atmos stack manifests

• The !terraform.output YAML function is used to read the outputs (remote state) of components directly in Atmos stack manifests

NOTE; You can use Atmos Stack Manifest Templating and Atmos YAML functions in the same stack configurations at the same time. Atmos processes the templates first, and then executes the YAML functions, allowing you to provide the parameters to the YAML functions dynamically.

Examples

components:
  terraform:
    component2:
      vars:
        # Handle the output of type list from the `atmos.Component` template function
        test_1: !template '{{ toJson (atmos.Component "component1" "plat-ue2-dev").outputs.test_list }}'

        # Handle the output of type map from the `atmos.Component` template function
        test_2: !template '{{ toJson (atmos.Component "component1" .stack).outputs.test_map }}'

        # Execute the shell script and assign the result to the `test_3` variable
        test_3: !exec echo 42

        # Execute the shell script to get the `test_label_id` output from the `component1` component in the stack `plat-ue2-dev`
        test_4: !exec atmos terraform output component1 -s plat-ue2-dev --skip-init -- -json test_label_id

        # Execute the shell script to get the `test_map` output from the `component1` component in the current stack
        test_5: !exec atmos terraform output component1 -s {{ .stack }} --skip-init -- -json test_map

        # Execute the shell script to get the `test_list` output from the `component1` component in the current stack
        test_6: !exec atmos terraform output component1 -s {{ .stack }} --skip-init -- -json test_list

        # Get the `test_label_id` output of type string from the `component1` component in the stack `plat-ue2-dev`
        test_7: !terraform.output component1 plat-ue2-dev test_label_id

        # Get the `test_label_id` output of type string from the `component1` component in the current stack
        test_8: !terraform.output component1 {{ .stack }} test_label_id

        # Get the `test_list` output of type list from the `component1` component in the current stack
        test_9: !terraform.output component1 {{ .stack }} test_list

        # Get the `test_map` output of type map from the `component1` component in the current stack
        test_10: !terraform.output component1 {{ .stack }} test_map

Summary by CodeRabbit

Release Notes

• New Features

  • Introduced new YAML functions: !exec, !template, and !terraform.output for enhanced stack manifest capabilities.
  • Added support for custom YAML tags processing in Atmos configurations.
  • Enhanced configuration options for Atlantis integration, allowing for more flexible setups.
  • Updated Atmos and Terraform versions in the Dockerfile for improved functionality.
  • Introduced new constants related to Atmos YAML functions for better configuration handling.

• Documentation Updates

  • Enhanced documentation for using remote state in Terraform components.
  • Updated guides for the atmos.Component function and the new YAML functions.
  • Clarified Atlantis integration setup options and workflows.
  • Improved explanations on handling outputs and using the new YAML functions.
  • Added documentation for new functions and updated existing guides for clarity.

• Dependency Updates

  • Upgraded various dependencies to their latest versions for improved performance and security.

[coderabbitai]

Actionable comments posted: 0

🧹 Outside diff range and nitpick comments (3)

website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx (3)

23-32: Consider enhancing argument descriptions with example values.

While the arguments are well-documented, adding example values would make it even clearer:

 <dl>
   <dt>`component`</dt>
-  <dd>Atmos component name</dd>
+  <dd>Atmos component name (e.g., `vpc`, `eks`, `rds`)</dd>

   <dt>`stack`</dt>
-  <dd>Atmos stack name</dd>
+  <dd>Atmos stack name (e.g., `plat-ue2-dev`, `plat-ue2-prod`)</dd>

   <dt>`output`</dt>
-  <dd>Terraform output</dd>
+  <dd>Terraform output (e.g., `vpc_id`, `subnet_ids`, `security_group_id`)</dd>
 </dl>

---

43-57: Consider adding example output formats.

The examples show different output types but could be more helpful with sample outputs:

 components:
   terraform:
     my_lambda_component:
       vars:
         vpc_config:
           # Output of type string
-          security_group_id: !terraform.output security-group/lambda {{ .stack }} id
+          # Returns: "sg-1234567890abcdef0"
+          security_group_id: !terraform.output security-group/lambda {{ .stack }} id
           # Output of type list
-          subnet_ids: !terraform.output vpc {{ .stack }} private_subnet_ids
+          # Returns: ["subnet-123...", "subnet-456..."]
+          subnet_ids: !terraform.output vpc {{ .stack }} private_subnet_ids
           # Output of type map
-          config_map: !terraform.output config {{ .stack }} config_map
+          # Returns: {"key1": "value1", "key2": "value2"}
+          config_map: !terraform.output config {{ .stack }} config_map

---

61-63: Consider adding a note about stack name validation.

It would be helpful to mention how Atmos validates the stack names and what happens if an invalid or non-existent stack is specified.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 5a4b2b3 and d03a3c1.

📒 Files selected for processing (1)

• website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx (1 hunks)

🧰 Additional context used

📓 Learnings (1)

website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx (2)

Learnt from: aknysh
PR: cloudposse/atmos#810
File: website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx:0-0
Timestamp: 2024-12-03T04:01:16.446Z
Learning: In the `terraform.output.mdx` documentation file (`website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx`), the cache invalidation and cache scope behavior for the `!terraform.output` function are already described.

Learnt from: aknysh
PR: cloudposse/atmos#810
File: website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx:104-110
Timestamp: 2024-12-03T03:49:30.395Z
Learning: In the documentation for `!terraform.output`, warnings about template variable availability are already covered in other sections, so no need to suggest adding them here.

🪛 LanguageTool

website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx

[typographical] ~133-~133: Consider using a typographic opening quote here.
Context: ... ``` :::tip Important By using the printf "%s-%s-%s" function, you are constructin...

(EN_QUOTES)

---

[style] ~178-~178: Using many exclamation marks might seem excessive (in this case: 11 exclamation marks for a text that’s 4082 characters long)
Context: ...e stacks or components. - When using !terraform.output with [`atmos describe ...

(EN_EXCESSIVE_EXCLAMATION)

🔇 Additional comments (4)

website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx (4)

1-13: LGTM! Clear and concise introduction.

The metadata and introduction effectively communicate the purpose of the !terraform.output function.

---

35-39: LGTM! Clear explanation of template processing order.

The tip effectively explains how template processing interacts with the !terraform.output function.

---

140-173: LGTM! Comprehensive caching explanation.

The caching behavior is well-documented with clear examples demonstrating the benefits.

---

174-185: LGTM! Thorough coverage of important considerations.

The considerations section effectively addresses security, performance, and operational aspects that users should be aware of.

🧰 Tools

🪛 LanguageTool

[style] ~178-~178: Using many exclamation marks might seem excessive (in this case: 11 exclamation marks for a text that’s 4082 characters long)
Context: ...e stacks or components. - When using !terraform.output with [`atmos describe ...

(EN_EXCESSIVE_EXCLAMATION)

[github-actions]

These changes were released in v1.111.0.