Update !terraform.state YAML function
#1383
Conversation
|
Important Cloud Posse Engineering Team Review RequiredThis pull request modifies files that require Cloud Posse's review. Please be patient, and a core maintainer will review your changes. To expedite this process, reach out to us on Slack in the |
📝 WalkthroughWalkthroughThis update covers dependency version bumps, enhancements to AWS S3 state reading logic with caching and retries, the addition of comprehensive YAML tag processing tests, and expanded documentation on using YQ expressions with Terraform state functions. Minor improvements include Dockerfile and workflow version updates, as well as new test fixtures for YAML function scenarios. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant App
participant AWS
participant S3
User->>App: Request Terraform state via !terraform.state tag
App->>App: Check S3 client cache (by region/role)
alt S3 client cached
App->>App: Use cached S3 client
else S3 client not cached
App->>AWS: Load AWS config (with assumeRoleDuration)
App->>App: Cache new S3 client
end
App->>S3: GetObject (with retries & timeout)
alt Success
S3-->>App: Return state file
App-->>User: Process and return requested state/output (with YQ transformation)
else NoSuchKey
S3-->>App: Not found
App-->>User: Return nil
else Other error
App->>S3: Retry up to 2 more times (with backoff)
App-->>User: Error if all retries fail
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~35 minutes Suggested labels
Suggested reviewers
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: .coderabbit.yaml ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (10)
🧰 Additional context used📓 Path-based instructions (4)website/**📄 CodeRabbit Inference Engine (.cursor/rules/atmos-rules.mdc)
Files:
**/*.go📄 CodeRabbit Inference Engine (.cursor/rules/atmos-rules.mdc)
Files:
**/*_test.go📄 CodeRabbit Inference Engine (.cursor/rules/atmos-rules.mdc)
Files:
go.mod📄 CodeRabbit Inference Engine (.cursor/rules/atmos-rules.mdc)
Files:
🧠 Learnings (11)📓 Common learningstests/fixtures/scenarios/atmos-terraform-state-yaml-function/stacks/deploy/nonprod.yaml (8)Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: Listener430 Learnt from: osterman Learnt from: aknysh Learnt from: aknysh Learnt from: haitham911 website/docs/integrations/atlantis.mdx (12)Learnt from: osterman Learnt from: aknysh Learnt from: Listener430 Learnt from: Listener430 Learnt from: Listener430 Learnt from: osterman Learnt from: RoseSecurity Learnt from: Cerebrovinny Learnt from: osterman Learnt from: aknysh Learnt from: haitham911 Learnt from: haitham911 internal/aws_utils/aws_utils_test.go (11)Learnt from: Listener430 Learnt from: CR Learnt from: CR Learnt from: CR Learnt from: CR Learnt from: CR Learnt from: CR Learnt from: samtholiya Learnt from: CR Learnt from: samtholiya Learnt from: Cerebrovinny website/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx (9)Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: osterman examples/quick-start-advanced/Dockerfile (10)Learnt from: osterman Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: Listener430 Learnt from: Listener430 Learnt from: Cerebrovinny Learnt from: aknysh Learnt from: osterman Learnt from: samtholiya internal/aws_utils/aws_utils.go (3)Learnt from: CR Learnt from: aknysh Learnt from: aknysh website/docs/core-concepts/stacks/yaml-functions/terraform.state.mdx (10)Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: osterman Learnt from: aknysh go.mod (15)Learnt from: CR Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: aknysh Learnt from: mcalhoun Learnt from: osterman Learnt from: osterman Learnt from: haitham911 Learnt from: haitham911 Learnt from: haitham911 Learnt from: aknysh Learnt from: CR Learnt from: CR Learnt from: CR internal/exec/yaml_func_utils_test.go (16)Learnt from: aknysh Learnt from: CR Learnt from: aknysh Learnt from: CR Learnt from: osterman Learnt from: Listener430 Learnt from: CR Learnt from: samtholiya Learnt from: CR Learnt from: haitham911 Learnt from: osterman Learnt from: haitham911 Learnt from: CR Learnt from: samtholiya Learnt from: aknysh Learnt from: aknysh internal/terraform_backend/terraform_backend_s3.go (3)Learnt from: Listener430 Learnt from: mcalhoun Learnt from: aknysh 🧬 Code Graph Analysis (3)tests/fixtures/scenarios/atmos-terraform-state-yaml-function/stacks/deploy/nonprod.yaml (1)
internal/aws_utils/aws_utils_test.go (1)
website/docs/core-concepts/stacks/yaml-functions/terraform.state.mdx (1)
🪛 LanguageToolwebsite/docs/core-concepts/stacks/yaml-functions/terraform.output.mdx[typographical] ~202-~202: Consider using a typographic close quote here. (EN_QUOTES) website/docs/core-concepts/stacks/yaml-functions/terraform.state.mdx[typographical] ~194-~194: Consider using a typographic close quote here. (EN_QUOTES) ⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
🔇 Additional comments (14)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1383 +/- ##
==========================================
+ Coverage 54.19% 54.24% +0.05%
==========================================
Files 259 259
Lines 27143 27164 +21
==========================================
+ Hits 14709 14736 +27
+ Misses 10729 10717 -12
- Partials 1705 1711 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
aknysh
added
no-release
|
These changes were released in v1.184.0. |
what
!terraform.stateYAML functionwhy
!terraform.stateYAML functions3backend, add retry logic when reading the state from the s3 bucket (in case of an error, retry two more times before returning the error)s3backend, increase the timeout from 5 sec to 30 sec (5 sec was not enough in some cases depending on network)s3backend, cache the S3 client (and don't assume role many times). When a stack uses many!terraform.statefunctions, create just one S3 client (assume the backend role if provided) and reuse it for all!terraform.statefunctions. This reduces the execution time and the API calls to AWS STSnotes
!terraform.stateAtmos YAML function #1363!terraform.stateYAML function #1374test
The improvements in this PR reduced the
!terraform.statefunctions execution time at least 2-3 times.For example, in a stack with 10
!terraform.outputfunctions, the following command executed for more than a minute:When switched to the
!terraform.stateYAML functions, the same command executes in just 15 seconds, from which the 10!terraform.stateYAML functions took 3-4 seconds, the rest isterraform planitself:With the improvements in this PR, the same command executes in 13 seconds, from which the 10
!terraform.stateYAML functions took 1-2 seconds:Summary by CodeRabbit