Skip to content

Expose ProcessTemplates and ProcessYamlFunctions flags in public API - #2161

Merged
Andriy Knysh (aknysh) merged 4 commits into
mainfrom
aknysh/updates-for-utils-provider-2
Mar 10, 2026
Merged

Andriy Knysh (aknysh) merged 4 commits into
mainfrom
aknysh/updates-for-utils-provider-2

Conversation

@aknysh

@aknysh Andriy Knysh (aknysh) commented Mar 10, 2026 •

Copy link
Copy Markdown
Member

what

  • Add optional ProcessTemplates and ProcessYamlFunctions controls to the ProcessComponentInStack and ProcessComponentFromContext public API functions in pkg/describe
  • Uses the functional options pattern: WithProcessTemplates(bool) and WithProcessYamlFunctions(bool)
  • Both functions now accept variadic ...ProcessOption — existing callers compile without changes
  • Both flags default to true when omitted, matching current behavior (templates processed if enabled in atmos.yaml, YAML functions always processed)

why

  • The public API hardcoded ProcessTemplates: true and ProcessYamlFunctions: true in processComponentInStackWithConfig, giving embedded consumers no way to disable processing
  • The Atmos CLI already has --process-templates and --process-functions flags, atmos.yaml has templates.settings.enabled, and stack imports have skip_templates_processing — the public API was the only entry point missing this control
  • terraform-provider-utils embeds Atmos and calls these functions from inside a Terraform provider plugin. When ProcessYamlFunctions is true, !terraform.output tags spawn child terraform init processes that conflict with the parent OpenTofu process's plugin cache, causing ETXTBSY ("text file busy") crashes on Linux
  • The provider only needs component backend config, workspace, and vars — it does not need resolved template or YAML function values
  • With this fix, the provider can pass WithProcessTemplates(false) and WithProcessYamlFunctions(false) to avoid the crash entirely

usage

// Disable both template and YAML function resolution
result, err := describe.ProcessComponentInStack(
    component, stack, configPath, basePath,
    describe.WithProcessTemplates(false),
    describe.WithProcessYamlFunctions(false),
)

// Same for ProcessComponentFromContext
result, err := describe.ProcessComponentFromContext(
    params,
    describe.WithProcessTemplates(false),
    describe.WithProcessYamlFunctions(false),
)

// Existing callers — no changes needed (both default to true)
result, err := describe.ProcessComponentInStack(component, stack, configPath, basePath)

tests

Test What It Verifies
TestProcessComponentInStackTemplatesDisabledOnly WithProcessTemplates(false) preserves raw Go template strings while YAML functions remain enabled
TestProcessComponentInStackTemplatesEnabledOnly WithProcessTemplates(true) resolves Go templates while YAML functions are disabled
TestProcessComponentInStackYamlFunctionsDisabledOnly WithProcessYamlFunctions(false) preserves raw YAML function tags while templates remain enabled
TestProcessComponentInStackYamlFunctionsEnabledOnly WithProcessYamlFunctions(true) resolves YAML function tags while templates are disabled
TestProcessComponentInStackBackwardCompatNoOptions Old 4-arg call (no options) still works and returns correct vars
TestProcessComponentFromContextWithProcessingDisabled ProcessComponentFromContext respects WithProcessTemplates(false) functional option

Each flag is tested independently against its own fixture (stack-templates for Go templates, atmos-template-yaml-function for YAML functions), proving the two flags are wired independently.

references

  • Previous related fix (restore public API): docs/fixes/2026-02-15-restore-component-processor-public-api.md
  • Previous related fix (serialize ReadDataSource): cloudposse/terraform-provider-utils#523

Summary by CodeRabbit

  • New Features

    • Added optional public flags (via functional options) to enable or disable template processing and YAML function processing for component description calls, preserving backward compatibility.
  • Documentation

    • Added usage guidance and compatibility notes for the new processing controls.
  • Tests

    • Added tests covering default behavior, enabling/disabling each processing flag, and consistency between different invocation paths.

…c API

Add optional processing controls to ProcessComponentInStack (variadic options)
and ProcessComponentFromContext (struct fields) so embedded consumers like
terraform-provider-utils can disable template and YAML function resolution.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@aknysh
Andriy Knysh (aknysh) requested a review from a team as a code owner March 10, 2026 18:02
@aknysh Andriy Knysh (aknysh) added the patch A minor, backward compatible change label Mar 10, 2026
@github-actions github-actions Bot added the size/m Medium size PR label Mar 10, 2026
@github-actions

github-actions Bot commented Mar 10, 2026 •

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues found.

Scanned Files

None

@coderabbitai

coderabbitai Bot commented Mar 10, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Adds a functional-options public API to control template and YAML-function processing for component description. Introduces an unexported processOptions struct, ProcessOption with WithProcessTemplates/WithProcessYamlFunctions, and updates ProcessComponentInStack and ProcessComponentFromContext to accept variadic options while preserving default behavior.

Changes

Cohort / File(s) Summary
Public API / Core logic
pkg/describe/component_processor.go
Add unexported processOptions, public ProcessOption type and constructors WithProcessTemplates/WithProcessYamlFunctions, plus defaultProcessOptions/applyProcessOptions. Change ProcessComponentInStack and ProcessComponentFromContext signatures to accept opts ...ProcessOption. Thread resolved options into processComponentInStackWithConfig and pass flags to ExecuteDescribeComponent.
Tests
pkg/describe/component_processor_test.go
Add tests for default behavior and explicit enabling/disabling of template and YAML-function processing for both stack and context entry points; includes backward-compatibility checks for no-options calls.
Docs
docs/fixes/2026-03-10-expose-process-templates-and-yaml-functions-flags-in-public-api.md
Document new functional-options API, defaults, usage examples, and backward-compatibility notes.

Sequence Diagram(s)

sequenceDiagram
  participant Caller as Caller (CLI / Consumer)
  participant API as ProcessComponentInStack / FromContext
  participant Core as processComponentInStackWithConfig
  participant Worker as ExecuteDescribeComponent
  participant Tmpl as TemplateProcessor
  participant Yaml as YamlFunctionProcessor

  rect rgba(0,128,255,0.5)
  Caller->>API: call with opts ...ProcessOption
  end

  API->>Core: applyProcessOptions(opts) -> resolved opts
  Core->>Worker: ExecuteDescribeComponent(..., ProcessTemplates: resolved.processTemplates, ProcessYamlFunctions: resolved.processYamlFunctions)
  alt ProcessTemplates enabled
    Worker->>Tmpl: run template processing
  end
  alt ProcessYamlFunctions enabled
    Worker->>Yaml: run YAML function processing
  end
  Worker-->>API: component map result
  API-->>Caller: return result
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Suggested labels

minor

Suggested reviewers

  • osterman
  • mcalhoun
🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly describes the main change: exposing ProcessTemplates and ProcessYamlFunctions flags in the public API, which is the core objective of the PR.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
  • 📝 Generate docstrings (stacked PR)
  • 📝 Generate docstrings (commit on current branch)
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch aknysh/updates-for-utils-provider-2

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
pkg/describe/component_processor_test.go (1)

14-22: Prefer testing the API behavior here, not the helper.

boolDefault is a tiny internal helper, so this mostly locks in implementation detail. The nil/default behavior that matters to callers is already exercised through ProcessComponentInStack and ProcessComponentFromContext, which is the better seam to test.

As per coding guidelines, "Test behavior, not implementation. Never test stub functions. Avoid tautological tests."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pkg/describe/component_processor_test.go` around lines 14 - 22, Remove the
unit test TestBoolDefault that directly exercises the internal helper
boolDefault and instead verify the same nil/default behavior through the public
APIs used by callers (e.g., add or extend tests for ProcessComponentInStack and
ProcessComponentFromContext to cover cases where the boolean pointer is nil and
when it is true/false), so tests assert observable behavior rather than the
internal boolDefault implementation; reference TestBoolDefault and boolDefault
to locate and delete the tautological test and add equivalent coverage via
ProcessComponentInStack/ProcessComponentFromContext tests.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/describe/component_processor_test.go`:
- Around line 311-419: The tests only exercise both-flags-false and nil
defaults, so add explicit unit tests that verify the flags are independently
wired by adding two new cases for ProcessComponentInStack and
ProcessComponentFromContext: one with ProcessTemplates=false and
ProcessYamlFunctions=true, and the inverse (ProcessTemplates=true and
ProcessYamlFunctions=false); use fixture components that include actual template
expressions and YAML functions (so behavior differs when each flag is toggled),
call ProcessComponentInStack (and ProcessComponentFromContext) with
ProcessComponentInStackOptions/params setting those pointer flags, and assert
that the output reflects only the enabled processor (e.g., template-resolved
values change when templates=true but YAML functions remain unresolved, and vice
versa) so the tests fail if a flag is ignored or swapped.

In `@pkg/describe/component_processor.go`:
- Around line 11-15: The public variadic pointer-bool config type
ProcessComponentInStackOptions should be replaced with a functional options API:
introduce an unexported options struct with concrete booleans defaulting to
true, define a ProcessOption (or similar) type like func(*options), and add
constructor helpers WithProcessTemplates(bool) and
WithProcessYamlFunctions(bool) that return ProcessOption; change the
ProcessComponentInStack signature to accept opts ...ProcessOption, apply
defaults then each option to mutate the internal options struct, and update all
callers to pass WithProcessTemplates(...) / WithProcessYamlFunctions(...)
instead of building ProcessComponentInStackOptions to eliminate *bool plumbing
and support future extensibility.
- Around line 62-66: The code reads only opts[0] and silently ignores additional
ProcessComponentInStackOptions entries; change this by explicitly handling
multiple option structs: either reject multiple options (check if len(opts) > 1
and return an error) or merge them deterministically (iterate over opts and set
processTemplates and processYamlFunctions from later entries or using a defined
precedence). Update the logic around the opts variable and the
processTemplates/processYamlFunctions assignments to perform the chosen
behavior, referencing the ProcessComponentInStackOptions type and the opts slice
so callers get a clear error or predictable merged result.

---

Nitpick comments:
In `@pkg/describe/component_processor_test.go`:
- Around line 14-22: Remove the unit test TestBoolDefault that directly
exercises the internal helper boolDefault and instead verify the same
nil/default behavior through the public APIs used by callers (e.g., add or
extend tests for ProcessComponentInStack and ProcessComponentFromContext to
cover cases where the boolean pointer is nil and when it is true/false), so
tests assert observable behavior rather than the internal boolDefault
implementation; reference TestBoolDefault and boolDefault to locate and delete
the tautological test and add equivalent coverage via
ProcessComponentInStack/ProcessComponentFromContext tests.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: e6721593-5134-4cc7-b3f8-6e1510e966a2

📥 Commits

Reviewing files that changed from the base of the PR and between 0b67828 and f7c963c.

📒 Files selected for processing (3)
  • docs/fixes/2026-03-10-expose-process-templates-and-yaml-functions-flags-in-public-api.md
  • pkg/describe/component_processor.go
  • pkg/describe/component_processor_test.go

Comment thread pkg/describe/component_processor_test.go Outdated
Comment thread pkg/describe/component_processor.go Outdated
Comment thread pkg/describe/component_processor.go Outdated
@codecov

codecov Bot commented Mar 10, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.16%. Comparing base (0b67828) to head (c663e47).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2161      +/-   ##
==========================================
- Coverage   77.17%   77.16%   -0.01%     
==========================================
  Files         951      951              
  Lines       90358    90375      +17     
==========================================
+ Hits        69737    69742       +5     
- Misses      16544    16553       +9     
- Partials     4077     4080       +3     
Flag Coverage Δ
unittests 77.16% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
pkg/describe/component_processor.go 88.40% <100.00%> (+3.79%) ⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Andriy Knysh (aknysh) and others added 2 commits March 10, 2026 14:54
Address PR review feedback:
- Replace ProcessComponentInStackOptions struct with functional options
  (WithProcessTemplates, WithProcessYamlFunctions)
- Remove TestBoolDefault (tests implementation, not behavior)
- Add tests that verify each flag independently using real fixtures
  (stack-templates for Go templates, atmos-template-yaml-function for YAML functions)
- Update fix doc to reflect functional options pattern

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@aknysh

Copy link
Copy Markdown
Member Author

CodeRabbit (@coderabbitai) full review please. If it looks good, please approve

@coderabbitai

coderabbitai Bot commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

Let me examine the code changes in detail before proceeding.


🏁 Script executed:

#!/bin/bash
# Read the main changed files
echo "=== pkg/describe/component_processor.go ==="
cat pkg/describe/component_processor.go

echo ""
echo "=== pkg/describe/component_processor_test.go ==="
cat pkg/describe/component_processor_test.go

echo ""
echo "=== docs/fixes/2026-03-10-expose-process-templates-and-yaml-functions-flags-in-public-api.md ==="
cat docs/fixes/2026-03-10-expose-process-templates-and-yaml-functions-flags-in-public-api.md

Length of output: 34612


@aknysh: The PR looks solid. Here's the full review:


✅ Overall Assessment: Approve

The implementation is clean, idiomatic Go, and correctly solves the embedded-consumer crash problem. No blocking issues found.


What's done well

  • Functional options pattern (ProcessOption, WithProcessTemplates, WithProcessYamlFunctions) is idiomatic and extensible — better than adding *bool fields to ComponentFromContextParams (which an earlier version apparently explored).
  • Backward compatibility is solid — both public functions use variadic opts ...ProcessOption, so all existing 4-argument callers compile and behave identically.
  • Defaults are correct — defaultProcessOptions() returns {true, true}, exactly matching the previous hardcoded values.
  • Tests are thorough — each flag is exercised independently against a targeted fixture (stack-templates for templates, atmos-template-yaml-function for YAML funcs), preventing cross-flag interference from masking bugs. The logOnFailure helper is a nice touch for CI debugging.
  • Documentation (docs/fixes/...) clearly explains the problem, the call chain that causes ETXTBSY, and the consumer upgrade path.

Minor nit (non-blocking)

perf.Track in the option constructors (WithProcessTemplates, WithProcessYamlFunctions): these functions just build a closure over a single bool assignment — they're sub-microsecond and produce noise in perf metrics without meaningful signal. Consider removing the defer perf.Track(...)() calls from the constructors (keeping them in ProcessComponentInStack and ProcessComponentFromContext where actual work happens):

// Before
func WithProcessTemplates(enabled bool) ProcessOption {
    defer perf.Track(nil, "describe.WithProcessTemplates")()
    return func(o *processOptions) {
        o.processTemplates = enabled
    }
}

// Suggested
func WithProcessTemplates(enabled bool) ProcessOption {
    return func(o *processOptions) {
        o.processTemplates = enabled
    }
}

Up to you — it doesn't affect correctness.


Approving now.

[approve]


🧠 Learnings used
Learnt from: Listener430
Repo: cloudposse/atmos PR: 934
File: tests/fixtures/scenarios/docs-generate/README.md.gotmpl:99-118
Timestamp: 2025-01-25T03:51:57.689Z
Learning: For the cloudposse/atmos repository, changes to template contents should be handled in dedicated PRs and are typically considered out of scope for PRs focused on other objectives.

Learnt from: osterman
Repo: cloudposse/atmos PR: 1599
File: internal/exec/terraform.go:394-402
Timestamp: 2025-10-10T23:51:36.597Z
Learning: In Atmos (internal/exec/terraform.go), when adding OpenTofu-specific flags like `--var-file` for `init`, do not gate them based on command name (e.g., checking if `info.Command == "tofu"` or `info.Command == "opentofu"`) because command names don't reliably indicate the actual binary being executed (symlinks, aliases). Instead, document the OpenTofu requirement in code comments and documentation, trusting users who enable the feature (e.g., `PassVars`) to ensure their terraform command points to an OpenTofu binary.

Learnt from: aknysh
Repo: cloudposse/atmos PR: 1327
File: cmd/terraform.go:111-117
Timestamp: 2025-06-23T02:14:30.937Z
Learning: In cmd/terraform.go, flags for the DescribeAffected function are added dynamically at runtime when info.Affected is true. This is intentional to avoid exposing internal flags like "file", "format", "verbose", "include-spacelift-admin-stacks", "include-settings", and "upload" in the terraform command interface, while still providing them for the shared DescribeAffected function used by both `atmos describe affected` and `atmos terraform apply --affected`.

Learnt from: osterman
Repo: cloudposse/atmos PR: 1686
File: docs/prd/tool-dependencies-integration.md:58-64
Timestamp: 2025-12-13T06:07:37.766Z
Learning: cloudposse/atmos: For PRD docs (docs/prd/*.md), markdownlint issues like MD040/MD010/MD034 can be handled in a separate documentation cleanup commit and should not block the current PR.

Learnt from: aknysh
Repo: cloudposse/atmos PR: 944
File: go.mod:206-206
Timestamp: 2025-01-17T00:18:57.769Z
Learning: For indirect dependencies with license compliance issues in the cloudposse/atmos repository, the team prefers to handle them in follow-up PRs rather than blocking the current changes, as these issues often require deeper investigation of the dependency tree.

Learnt from: RoseSecurity
Repo: cloudposse/atmos PR: 1448
File: cmd/ansible.go:26-28
Timestamp: 2025-09-05T14:57:37.360Z
Learning: The Atmos codebase uses a consistent pattern for commands that delegate to external tools: `PersistentFlags().Bool("", false, doubleDashHint)` where doubleDashHint provides help text about using double dashes to separate Atmos options from native command arguments. This pattern is used across terraform, packer, helmfile, atlantis, aws, and ansible commands.

Learnt from: Listener430
Repo: cloudposse/atmos PR: 825
File: internal/exec/helmfile_generate_varfile.go:28-31
Timestamp: 2024-12-07T16:16:13.038Z
Learning: In `internal/exec/helmfile_generate_varfile.go`, the `--help` command (`./atmos helmfile generate varfile --help`) works correctly without requiring stack configurations, and the only change needed was to make `ProcessCommandLineArgs` exportable by capitalizing its name.

Learnt from: CR
Repo: cloudposse/atmos PR: 0
File: agent-skills/AGENTS.md:0-0
Timestamp: 2026-03-03T16:29:20.024Z
Learning: Applies to agent-skills/**/{stacks,components}/**/*.{yaml,tpl} : Use !terraform.output YAML function or {{ atmos.Component }} Go template for cross-stack component references instead of hardcoding values

Learnt from: CR
Repo: cloudposse/atmos PR: 0
File: agent-skills/AGENTS.md:0-0
Timestamp: 2026-03-03T16:29:20.024Z
Learning: Applies to agent-skills/**/{stacks,workflows,components}/**/*.{yaml,tpl} : Use Go templates with Sprig/Gomplate functions, atmos.Component, and atmos.GomplateDatasource for dynamic configuration generation in stack and workflow files

coderabbitai[bot]
coderabbitai Bot previously approved these changes Mar 10, 2026
WithProcessTemplates and WithProcessYamlFunctions just build a closure
over a single bool — sub-microsecond with no meaningful perf signal.
Suppress lintroller with nolint directive instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pkg/describe/component_processor.go`:
- Around line 159-161: The options (applyProcessOptions) are being resolved
after stackNameTemplate is rendered, so WithProcessTemplates(false) won't
prevent template execution for the stack name; move the resolution/application
of process options earlier (call applyProcessOptions(opts) and ensure
WithProcessTemplates is applied to the context path) before any use of
stackNameTemplate/stack-name branching (references: applyProcessOptions,
WithProcessTemplates, stackNameTemplate, ProcessComponentFromContext,
processComponentInStackWithConfig) so the template-processing flag takes effect
for the stack-name branch; alternatively, if you prefer not to reorder, update
the public contract/docs to state that the flag only affects component body
processing.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 9e1eca3f-d7a1-4856-931a-5d10a4dd65cc

📥 Commits

Reviewing files that changed from the base of the PR and between 78ebc8a and c663e47.

📒 Files selected for processing (1)
  • pkg/describe/component_processor.go

Comment thread pkg/describe/component_processor.go
@aknysh
Andriy Knysh (aknysh) merged commit 66ac17c into main Mar 10, 2026
55 checks passed
@aknysh
Andriy Knysh (aknysh) deleted the aknysh/updates-for-utils-provider-2 branch March 10, 2026 20:14
@github-actions

Copy link
Copy Markdown

These changes were released in v1.209.0.

1 similar comment
@github-actions

Copy link
Copy Markdown

These changes were released in v1.209.0.

@github-actions

Copy link
Copy Markdown

These changes were released in v1.209.0.

@github-actions

Copy link
Copy Markdown

These changes were released in v1.210.0-test.8.

@github-actions

Copy link
Copy Markdown

These changes were released in v1.210.0-test.10.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

patch A minor, backward compatible change size/m Medium size PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants