Skip to content

fix(output): render concurrent carriage-return updates safely - #2860

Merged
Andriy Knysh (aknysh) merged 20 commits into
cloudposse:mainfrom
zack-is-cool:fix/concurrent-terminal-output
Aug 5, 2026
Merged

Andriy Knysh (aknysh) merged 20 commits into
cloudposse:mainfrom
zack-is-cool:fix/concurrent-terminal-output

Conversation

@zack-is-cool

@zack-is-cool zack-is-cool commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

what

  • Keep concurrent component output readable when underlying tools emit carriage-return progress updates.
  • Serialize prefixed stdout and stderr writes to their shared terminal.
  • Disable animated output-lookup spinners while concurrent Terraform work is running.
  • Add regression coverage for carriage-return handling and nested spinner suppression.

why

  • Concurrent writers and terminal redraw controls can otherwise reposition the cursor or interleave output, corrupting rendered lines.

references

validation

  • go test ./pkg/scheduler/adapters ./pkg/io -count=1
  • go test ./pkg/terraform/output -run '^TestSuppressSpinnersRestoresNestedScopes$' -count=1
  • pre-commit run --files pkg/scheduler/adapters/terraform.go pkg/terraform/output/executor_utils.go pkg/terraform/output/spinner.go pkg/terraform/output/spinner_test.go

Summary by CodeRabbit

  • Bug Fixes

    • Improved line-prefixed output for Unix, Windows, and standalone carriage-return line endings.
    • Prevented partial lines from being lost during flushing or after write errors.
    • Prevented interleaving of Terraform standard output and error output during concurrent execution.
  • Improvements

    • Suppressed transient spinners and provisioning messages during streamed or concurrent output.
    • Improved spinner cleanup across successful runs, errors, and nested operations.
    • Standardized carriage-return output as newline-delimited, prefixed lines.
    • Routed hook output consistently through the appropriate component streams.
    • Improved synchronization for grouped and concurrent Terraform output.

@zack-is-cool
zack-is-cool requested a review from a team as a code owner August 3, 2026 21:48
@atmos-pro

atmos-pro Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Tip

Atmos Pro  

No affected stacks workflow was detected for this pull request.
If this is expected, no action is needed.
Learn More. Ask AI.

@github-actions github-actions Bot added the size/s Small size PR label Aug 3, 2026
@coderabbitai

coderabbitai Bot commented Aug 3, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

LinePrefixWriter normalizes newline variants. Concurrent Terraform execution suppresses transient UI output, routes hook output through component writers, and serializes prefixed streams.

Changes

Terraform output handling

Layer / File(s) Summary
Line delimiter handling
pkg/io/line_prefix_writer.go, pkg/io/line_prefix_writer_test.go
The writer detects and normalizes newline, carriage-return, and CRLF delimiters. Tests cover split delimiters, trailing data, and retry after write errors.
Spinner suppression and lifecycle
pkg/terraform/output/*
Spinner suppression uses synchronized reference counting. Lookup and provisioning paths conditionally emit transient UI output.
Suppressed provisioning output
pkg/provisioner/workdir/*, pkg/provisioner/source/*, pkg/terraform/output/executor_workdir.go
Suppression propagates through workdir provisioning and source vending. Status and warning output is omitted when suppression is active.
Writer-aware hook execution
pkg/schema/schema.go, pkg/hooks/*, cmd/helmfile/helmfile.go, cmd/terraform/utils.go
Component hooks and subprocesses accept configured stdout and stderr writers while retaining legacy hook behavior.
Concurrent Terraform stream coordination
pkg/scheduler/adapters/terraform.go, pkg/scheduler/adapters/terraform_test.go
Concurrent executions suppress spinners, pass per-node writers to hooks, and share one mutex for prefixed stdout and stderr streams.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

Suggested reviewers: aknysh, osterman

Sequence Diagram(s)

sequenceDiagram
  participant TerraformExecutions
  participant TerraformAdapter
  participant SpinnerOutput
  participant Provisioner
  participant ComponentHooks
  participant PrefixedWriters
  TerraformExecutions->>TerraformAdapter: Run concurrent invocations
  TerraformAdapter->>SpinnerOutput: Suppress spinners
  TerraformAdapter->>Provisioner: Provision workdirs with suppressed output
  TerraformAdapter->>ComponentHooks: Pass component stdout and stderr writers
  ComponentHooks->>PrefixedWriters: Write hook output
  TerraformAdapter->>PrefixedWriters: Stream Terraform output
  PrefixedWriters->>PrefixedWriters: Normalize carriage-return lines
  TerraformAdapter->>SpinnerOutput: Restore spinner state
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 26.32% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main fix for safe concurrent carriage-return rendering.
Linked Issues check ✅ Passed The changes address issue #2859 by normalizing carriage returns, serializing output, suppressing transient UI, and preserving prefixed component output.
Out of Scope Changes check ✅ Passed The changes support the linked issue by covering related UI suppression, hook output routing, serialization, and regression tests.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]
coderabbitai Bot previously approved these changes Aug 3, 2026
@github-actions github-actions Bot added size/m Medium size PR and removed size/s Small size PR labels Aug 3, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/terraform/output/spinner_test.go`:
- Around line 19-31: Update TestSuppressSpinnersRestoresNestedScopes to register
both restoreOuter and restoreInner with t.Cleanup immediately after creating
them, ensuring cleanup runs after assertion failures. Invoke one restoration
function twice and verify suppression remains enabled while the other scope is
active, then restore the remaining scope and verify spinnersSuppressed() is
false.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e3d6ddba-69f8-4f0e-af45-6e4c97dc7314

📥 Commits

Reviewing files that changed from the base of the PR and between 82271c1 and d5d22cb.

📒 Files selected for processing (4)
  • pkg/scheduler/adapters/terraform.go
  • pkg/terraform/output/executor_utils.go
  • pkg/terraform/output/spinner.go
  • pkg/terraform/output/spinner_test.go

Comment thread pkg/terraform/output/spinner_test.go

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/terraform/output/executor_utils.go`:
- Around line 105-116: Update finishSpinner to call clearSpinnerLine() during
cleanup instead of using the captured clearsLine flag and directly invoking
ui.ClearLine(), so suppression state is checked at cleanup time. Add a
regression test covering suppression beginning after spinner creation but before
the returned cleanup function runs.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 23e55a89-3bf3-49f6-9450-6fc75310ae83

📥 Commits

Reviewing files that changed from the base of the PR and between d5d22cb and c277448.

📒 Files selected for processing (4)
  • pkg/terraform/output/executor.go
  • pkg/terraform/output/executor_test.go
  • pkg/terraform/output/executor_utils.go
  • pkg/terraform/output/executor_workdir.go

Comment thread pkg/terraform/output/executor_utils.go Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/scheduler/adapters/terraform_test.go`:
- Around line 27-32: Remove the duplicate adapterWriterFunc type and its Write
method from the test file, and reuse the existing package-level
adapterWriterFunc defined in terraform.go.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 153502a1-d45b-4c1d-a438-6bcf5c9eeb60

📥 Commits

Reviewing files that changed from the base of the PR and between c277448 and c84d434.

📒 Files selected for processing (2)
  • pkg/scheduler/adapters/terraform.go
  • pkg/scheduler/adapters/terraform_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • pkg/scheduler/adapters/terraform.go

Comment thread pkg/scheduler/adapters/terraform_test.go Outdated
@zack-is-cool

Copy link
Copy Markdown
Contributor Author

CodeRabbit (@coderabbitai) full review

@coderabbitai

coderabbitai Bot commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/terraform/output/executor_utils.go`:
- Around line 118-122: Update clearSpinnerLine to hold spinnerSuppression while
checking the suppression count and calling ui.ClearLine, rather than using
spinnersSuppressed() which releases the lock before clearing. Preserve the
existing behavior of skipping the clear when suppression is active and ensure
SuppressSpinners cannot interleave between the check and line clear.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: a0aa78bf-3282-4899-9d79-7b20fc4eea61

📥 Commits

Reviewing files that changed from the base of the PR and between e299b51 and 6ef9368.

📒 Files selected for processing (10)
  • pkg/io/line_prefix_writer.go
  • pkg/io/line_prefix_writer_test.go
  • pkg/scheduler/adapters/terraform.go
  • pkg/scheduler/adapters/terraform_test.go
  • pkg/terraform/output/executor.go
  • pkg/terraform/output/executor_test.go
  • pkg/terraform/output/executor_utils.go
  • pkg/terraform/output/executor_workdir.go
  • pkg/terraform/output/spinner.go
  • pkg/terraform/output/spinner_test.go

Comment thread pkg/terraform/output/executor_utils.go
@zack-is-cool

Copy link
Copy Markdown
Contributor Author

CodeRabbit (@coderabbitai) full review

@coderabbitai

coderabbitai Bot commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Full review finished.


Your included review limit is currently reached under our Fair Usage Limits Policy. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 29 minutes.

@osterman

Copy link
Copy Markdown
Member

CodeRabbit (@coderabbitai) my concern is that Atmos should be automatically downgrading things like spinners when running in parallel mode so that we don't need to worry about carriage returns like this. I suspect this was always a problem, but a recent pull request that was merged fixed our TTY detection, and that's why we see this now. While I agree we should gracefully handle carriage returns from sub-processes that are not Atmos, if it's Atmos itself, then it should just not be emitting the spinners, for example. With that in mind, is this the correct fix?

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- changed files ---'
git diff --name-only HEAD^ HEAD

printf '%s\n' '--- outlines ---'
ast-grep outline pkg/scheduler/adapters/terraform.go --items all
ast-grep outline pkg/terraform/output/spinner.go --items all
ast-grep outline pkg/terraform/output/executor_utils.go --items all
ast-grep outline pkg/io/line_prefix_writer.go --items all

printf '%s\n' '--- scheduler adapter relevant code ---'
rg -n -C 12 'SuppressSpinners|newSerializedWriter|LinePrefixWriter|parallel|concurrent|mutex|Mutex' pkg/scheduler/adapters/terraform.go pkg/scheduler/adapters/terraform_test.go

printf '%s\n' '--- spinner and executor relevant code ---'
rg -n -C 10 'SuppressSpinners|spinnersSuppressed|startSpinnerOrLog|finishSpinner|clearSpinnerLine|StartSpinner|ClearLine' \
  pkg/terraform/output/spinner.go \
  pkg/terraform/output/executor.go \
  pkg/terraform/output/executor_utils.go \
  pkg/terraform/output/executor_workdir.go \
  pkg/terraform/output/executor_test.go \
  pkg/terraform/output/spinner_test.go

printf '%s\n' '--- line writer relevant code ---'
rg -n -C 12 'func \(.*LinePrefixWriter.*\) (Write|Flush)|\\r|\\n' \
  pkg/io/line_prefix_writer.go \
  pkg/io/line_prefix_writer_test.go

Length of output: 90323


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- ExecuteTerraform control flow ---'
sed -n '133,235p' pkg/scheduler/adapters/terraform.go

printf '%s\n' '--- output lifecycle and dispatch path ---'
sed -n '608,715p' pkg/scheduler/adapters/terraform.go
sed -n '1238,1463p' pkg/scheduler/adapters/terraform.go

printf '%s\n' '--- all spinner construction and suppression call sites ---'
rg -n -C 4 'NewSpinner\s*\(|RunSpinner\s*\(|SuppressSpinners\s*\(|spinnersSuppressed\s*\(' --type go .

printf '%s\n' '--- adapter tests that exercise suppression or output lookups ---'
rg -n -C 8 'SuppressSpinners|spinnersSuppressed|GetOutput|GetAllOutputs|MaxConcurrency:\s*[2-9]|MaxConcurrency:\s*[1-9][0-9]' \
  pkg/scheduler/adapters/terraform_test.go \
  pkg/terraform/output/executor_test.go \
  pkg/terraform/output/spinner_test.go

printf '%s\n' '--- complete PR diff from merge base ---'
base="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD main 2>/dev/null || true)"
if [ -n "$base" ]; then
  git diff --stat "$base"..HEAD
  git diff -- "$base"..HEAD -- pkg/scheduler/adapters/terraform.go pkg/terraform/output/spinner.go pkg/terraform/output/executor_utils.go pkg/io/line_prefix_writer.go
else
  echo 'Merge base is unavailable.'
fi

Length of output: 102371


@osterman Yes. This is the correct design.

The fix now has two layers:

  1. Atmos-owned transient UI is disabled for parallel Terraform execution.
    ExecuteTerraform calls tfoutput.SuppressSpinners() when effective concurrency is greater than one. The suppression remains active through scheduler execution and result handling. startSpinnerOrLog then does not create output-lookup spinners. The related transient output and ui.ClearLine() calls are also disabled.

  2. Subprocess output remains defensive.
    Terraform, OpenTofu, providers, and other child processes can still emit \r progress updates. LinePrefixWriter converts those records to newline-delimited, prefixed output and serializes stdout and stderr writes. This prevents cursor repositioning and cross-component overwrite even when the output is not produced by Atmos.

So, carriage-return normalization should not be the primary solution for Atmos spinners. It is the required safety boundary for child-process output. The spinner suppression is the primary solution for Atmos output.

I found one coverage gap. The tests verify SuppressSpinners() and its nested cleanup behavior, but they do not verify that ExecuteTerraform activates suppression when MaxConcurrency > 1 and restores it after completion. I recommend adding that adapter-level regression test. It will protect the policy that parallel execution must downgrade transient UI.

🐇🔎


✏️ Learnings added
Learnt from: osterman
URL: https://github.com/cloudposse/atmos/pull/2860

Timestamp: 2026-08-04T02:44:16.181Z
Learning: For concurrent Terraform execution in `pkg/scheduler/adapters/terraform.go`, Atmos must suppress Atmos-owned transient UI such as output-lookup spinners when effective `MaxConcurrency` is greater than one. Carriage-return normalization in `pkg/io/line_prefix_writer.go` remains a defensive mechanism for Terraform/OpenTofu and other subprocess output, not the primary mechanism for Atmos UI.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

You are interacting with an AI system.

@codecov

codecov Bot commented Aug 4, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.84530% with 22 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@c14ce82). Learn more about missing BASE report.

Files with missing lines Patch % Lines
pkg/provisioner/source/provision_hook.go 36.36% 6 Missing and 1 partial ⚠️
pkg/terraform/output/executor.go 71.42% 4 Missing ⚠️
pkg/hooks/command_engine.go 83.33% 2 Missing and 1 partial ⚠️
pkg/io/line_prefix_writer.go 87.50% 1 Missing and 1 partial ⚠️
pkg/provisioner/workdir/workdir.go 91.30% 1 Missing and 1 partial ⚠️
pkg/scheduler/adapters/terraform.go 89.47% 1 Missing and 1 partial ⚠️
pkg/terraform/output/executor_utils.go 91.30% 1 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #2860   +/-   ##
=======================================
  Coverage        ?   82.75%           
=======================================
  Files           ?     1860           
  Lines           ?   180337           
  Branches        ?        0           
=======================================
  Hits            ?   149243           
  Misses          ?    23309           
  Partials        ?     7785           
Flag Coverage Δ
unittests 82.75% <87.84%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
cmd/helmfile/helmfile.go 98.68% <100.00%> (ø)
cmd/terraform/utils.go 71.55% <100.00%> (ø)
pkg/hooks/hooks.go 84.46% <100.00%> (ø)
pkg/hooks/kind.go 100.00% <ø> (ø)
pkg/hooks/kinds/tfmigrate/kind.go 92.15% <100.00%> (ø)
pkg/schema/schema.go 86.42% <ø> (ø)
pkg/terraform/output/executor_workdir.go 95.74% <100.00%> (ø)
pkg/terraform/output/spinner.go 91.93% <100.00%> (ø)
pkg/io/line_prefix_writer.go 96.92% <87.50%> (ø)
pkg/provisioner/workdir/workdir.go 95.54% <91.30%> (ø)
... and 5 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@osterman Erik Osterman (Cloud Posse) (osterman) added the patch A minor, backward compatible change label Aug 4, 2026
@zack-is-cool

Copy link
Copy Markdown
Contributor Author

Added the requested adapter-level regression in 0ecff7a. It verifies that a MaxConcurrency: 2 run activates spinner suppression while nodes execute and restores it when ExecuteTerraform returns.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/hooks/command_engine_test.go`:
- Around line 191-222: The test
TestCommandEngine_RoutesSubprocessOutputToContextWriters currently exercises
only stdout routing. Update its child-process fixture to emit a distinct stderr
value, configure the corresponding test environment input, and assert that the
value appears only in the context’s stderr buffer while stdout remains
unchanged, providing coverage for both writer paths.

In `@pkg/scheduler/adapters/terraform_test.go`:
- Around line 374-386: Update testNodeHooks.BeforeWithWriters and the
corresponding hook-writing path so n.mu protects only state recording and error
lookup, not fmt.Fprint; release the lock before writing and add separate
synchronization that coordinates fragmented writes from concurrent hooks through
their node writers. Strengthen TestExecuteTerraformConcurrentHooksUseNodeWriters
with behavior-focused assertions proving concurrent writes contend correctly and
cannot pass merely because shared writer synchronization serializes them.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7bf9eed3-5c8a-4d25-ad68-43d168f70533

📥 Commits

Reviewing files that changed from the base of the PR and between af63a4c and 8cf80fb.

📒 Files selected for processing (16)
  • cmd/helmfile/helmfile.go
  • cmd/terraform/utils.go
  • pkg/hooks/command_engine.go
  • pkg/hooks/command_engine_test.go
  • pkg/hooks/hooks.go
  • pkg/hooks/kind.go
  • pkg/hooks/kinds/tfmigrate/kind.go
  • pkg/io/line_prefix_writer.go
  • pkg/io/line_prefix_writer_test.go
  • pkg/scheduler/adapters/terraform.go
  • pkg/scheduler/adapters/terraform_test.go
  • pkg/schema/schema.go
  • pkg/terraform/output/executor.go
  • pkg/terraform/output/executor_utils.go
  • pkg/terraform/output/executor_workdir.go
  • pkg/terraform/output/spinner_test.go
💤 Files with no reviewable changes (1)
  • pkg/terraform/output/executor.go
🚧 Files skipped from review as they are similar to previous changes (4)
  • pkg/terraform/output/executor_workdir.go
  • pkg/terraform/output/executor_utils.go
  • pkg/io/line_prefix_writer_test.go
  • pkg/io/line_prefix_writer.go

Comment thread pkg/hooks/command_engine_test.go
Comment thread pkg/scheduler/adapters/terraform_test.go Outdated
coderabbitai[bot]
coderabbitai Bot previously approved these changes Aug 4, 2026
@zack-is-cool

Copy link
Copy Markdown
Contributor Author

CodeRabbit (@coderabbitai) Anything else?

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

base="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD main 2>/dev/null)"
printf '%s\n' '--- PR commits and changed files ---'
git log --oneline "${base}..HEAD"
git diff --name-only "${base}..HEAD"

printf '%s\n' '--- final hook-output commit ---'
git show --format=fuller --stat 8cf80fbd0
git show --format= --unified=80 8cf80fbd0 -- \
  pkg/scheduler/adapters/terraform.go \
  pkg/scheduler/adapters/terraform_test.go \
  pkg/hooks \
  pkg/schema/schema.go \
  cmd/terraform/utils.go \
  cmd/helmfile/helmfile.go \
  pkg/io/line_prefix_writer.go \
  pkg/io/line_prefix_writer_test.go

printf '%s\n' '--- relevant outlines ---'
for f in \
  pkg/scheduler/adapters/terraform.go \
  pkg/scheduler/adapters/terraform_test.go \
  pkg/hooks/hooks.go \
  pkg/hooks/command_engine.go \
  pkg/hooks/kind.go \
  pkg/hooks/kinds/tfmigrate/kind.go \
  pkg/schema/schema.go \
  cmd/terraform/utils.go \
  cmd/helmfile/helmfile.go \
  pkg/io/line_prefix_writer.go \
  pkg/io/line_prefix_writer_test.go
do
  if [ -f "$f" ]; then
    echo "--- $f ---"
    ast-grep outline "$f" --items all
  fi
done

printf '%s\n' '--- hook contracts, implementations, and writer propagation ---'
rg -n -C 12 --type go \
  'ComponentNodeHooksWithOutput|ComponentNodeHookWriters|BeforeWithWriters|AfterWithWriters|RunPerComponentHooksOptions|ExecContext|subprocessPrep|Stdout:|Stderr:' \
  pkg cmd

printf '%s\n' '--- serialized writer and line handling ---'
sed -n '1,260p' pkg/io/line_prefix_writer.go
sed -n '1,380p' pkg/io/line_prefix_writer_test.go
rg -n -C 8 --type go \
  'newSerializedWriter|NewLinePrefixWriter|\.Flush\(\)|outputMu|nodeWriters|LinePrefixWriter' \
  pkg/scheduler/adapters pkg/io

printf '%s\n' '--- direct output from hook and scheduler paths ---'
rg -n -C 8 --type go \
  'os\.(Stdout|Stderr)|fmt\.(Fprint|Print)|ui\.[A-Za-z0-9_]+|iolib\.(UI|Data)|process\.Streams|renderTerminal' \
  pkg/hooks pkg/scheduler/adapters cmd/terraform cmd/helmfile

Length of output: 151407


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- lifecycle hook dispatch ---'
sed -n '640,715p' pkg/scheduler/adapters/terraform.go
rg -n -C 20 --type go \
  'func \(.*\) run(Before|After)NodeHooks|BeforeWithWriters|AfterWithWriters|RunPerComponentHooks' \
  pkg/scheduler/adapters/terraform.go \
  cmd/terraform/utils.go \
  cmd/helmfile/helmfile.go

printf '%s\n' '--- writer option propagation and resolved-hook execution ---'
rg -n -C 20 --type go \
  'type RunPerComponentHooksOptions|func RunPerComponentHooks|func \(.*Hooks.*\) runResolvedHook|ExecContext\{' \
  pkg/hooks/hooks.go
sed -n '80,145p' pkg/hooks/command_engine.go
sed -n '240,370p' pkg/hooks/command_engine.go

printf '%s\n' '--- structured and markdown hook output configuration ---'
rg -n -C 12 --type go \
  'CaptureOutput|capture.output|FormatMarkdown|Format.*Markdown|Summary|Artifact|renderTerminal' \
  pkg/hooks cmd/terraform pkg/scheduler/adapters

printf '%s\n' '--- tests for concurrent node hooks and captured hook output ---'
sed -n '450,560p' pkg/scheduler/adapters/terraform_test.go
rg -n -C 12 --type go \
  'RunPerComponentHooks|CaptureOutput|FormatMarkdown|renderTerminal|Stdout.*Stderr' \
  pkg/hooks/*_test.go pkg/hooks/**/*_test.go pkg/scheduler/adapters/terraform_test.go

Length of output: 246793


@zack-is-cool Yes. I found one remaining blocking gap.

P1 — Serialize structured hook summaries during concurrent execution

pkg/hooks/command_engine.go correctly sends subprocess stdout and stderr to ExecContext.Stdout and ExecContext.Stderr. However, renderTerminal bypasses these writers at Lines 349-355. It calls ui.Writeln and ui.MarkdownMessage directly.

Built-in scanner hooks and Infracost hooks can produce a non-empty Summary.Body. When concurrent Terraform nodes run these hooks, their rendered Markdown bypasses terraformOutput.outputMu. The output is not component-prefixed. It can interleave with prefixed Terraform and hook subprocess output.

Route renderTerminal output through the supplied ExecContext writer when one is set. Preserve the existing UI renderer for sequential execution. Add a regression test with two concurrent nodes that emit structured hook summaries.

The subprocess stream path is now correct. The structured hook-summary path remains outside the serialized path.

You are interacting with an AI system.

@osterman

Copy link
Copy Markdown
Member

CodeRabbit (@coderabbitai) why do you keep finding "one remaining blocking gap." Can you just give it to us all at once?

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

base="$(git merge-base HEAD origin/main 2>/dev/null || git merge-base HEAD main 2>/dev/null)"

printf '%s\n' '--- PR files and complete diff ---'
git diff --name-only "${base}..HEAD"
git diff --find-renames --unified=70 "${base}..HEAD" -- \
  pkg/scheduler/adapters \
  pkg/terraform/output \
  pkg/provisioner/workdir \
  pkg/provisioner/source \
  pkg/hooks \
  pkg/io/line_prefix_writer.go \
  pkg/schema/schema.go \
  cmd/terraform \
  cmd/helmfile

printf '%s\n' '--- maps of concurrent scheduler and output-relevant packages ---'
for path in \
  pkg/scheduler/adapters/terraform.go \
  pkg/terraform/output \
  pkg/provisioner/workdir \
  pkg/provisioner/source \
  pkg/hooks \
  pkg/io/line_prefix_writer.go \
  cmd/terraform/utils.go \
  cmd/helmfile/helmfile.go
do
  if [ -e "$path" ]; then
    echo "--- $path ---"
    ast-grep outline "$path" --items all || true
  fi
done

printf '%s\n' '--- all direct terminal/UI write sites in reachable packages ---'
rg -n -C 8 --type go \
  '(ui\.[A-Za-z0-9_]+\s*\(|iolib\.(UI|Data)\.(Write|WriteString)\s*\(|os\.(Stdout|Stderr)\.(Write|WriteString)\s*\(|fmt\.Fprint[a-z]*\s*\(|process\.Streams\s*\{|Stdout:|Stderr:)' \
  pkg/scheduler/adapters pkg/terraform/output pkg/provisioner/workdir pkg/provisioner/source pkg/hooks cmd/terraform cmd/helmfile

printf '%s\n' '--- suppression and component-writer propagation ---'
rg -n -C 12 --type go \
  '(SuppressSpinners|spinnersSuppressed|WithOutputSuppressed|OutputSuppressed|ComponentNodeHooksWithOutput|BeforeWithWriters|AfterWithWriters|ComponentNodeHookWriters|RunPerComponentHooksOptions|ExecContext|newSerializedWriter|NewLinePrefixWriter|renderTerminal)' \
  pkg/scheduler/adapters pkg/terraform/output pkg/provisioner/workdir pkg/provisioner/source pkg/hooks pkg/schema cmd/terraform cmd/helmfile

printf '%s\n' '--- complete implementations at output boundaries ---'
sed -n '120,250p' pkg/scheduler/adapters/terraform.go
sed -n '620,735p' pkg/scheduler/adapters/terraform.go
sed -n '1220,1490p' pkg/scheduler/adapters/terraform.go
sed -n '1,430p' pkg/hooks/command_engine.go
sed -n '1,280p' pkg/hooks/hooks.go
sed -n '1,260p' pkg/hooks/kind.go
sed -n '1,300p' pkg/hooks/kinds/tfmigrate/kind.go
sed -n '1,260p' pkg/io/line_prefix_writer.go

Length of output: 115411


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- non-test hook engine implementations and direct terminal writes ---'
rg -n -C 6 --glob '!**/*_test.go' --type go \
  'func \(.*\) Run\(ctx \*hooks?\.?ExecContext\)|func \(.*\) Run\(ctx \*ExecContext\)|ui\.(Writeln|MarkdownMessage|Infof?|Successf?|Warningf?|Errorf?|ClearLine)\s*\(|os\.(Stdout|Stderr)|fmt\.(Print|Fprint)' \
  pkg/hooks

printf '%s\n' '--- registered hook kinds and their engines ---'
rg -n -C 5 --glob '!**/*_test.go' --type go \
  'RegisterKind\s*\(|Engine:\s*|type Engine struct' \
  pkg/hooks

printf '%s\n' '--- concurrent dispatcher paths and direct UI writes ---'
rg -n -C 12 --glob '!**/*_test.go' --type go \
  'func \(d \*TerraformDispatcher\) Dispatch|ui\.[A-Za-z0-9_]+\s*\(|runBeforeNodeHooks|runAfterNodeHooks|nodeWriters|finishNode|writeTerraformSummary' \
  pkg/scheduler/adapters/terraform.go

printf '%s\n' '--- UI and CI log-group output implementations ---'
ast-grep outline pkg/ui --items all
rg -n -C 10 --glob '!**/*_test.go' --type go \
  'func (Writeln|MarkdownMessage|Info|Infof|Success|Successf|Warning|Warningf|Error|Errorf|ClearLine)\s*\(|func (Group|StartLogGroup)\s*\(' \
  pkg/ui pkg/ci

printf '%s\n' '--- focused source slices ---'
sed -n '300,390p' pkg/hooks/command_engine.go
sed -n '1,240p' pkg/hooks/step_engine.go
sed -n '1,220p' pkg/hooks/kinds/tfmigrate/kind.go
sed -n '650,730p' pkg/scheduler/adapters/terraform.go
sed -n '1,220p' pkg/scheduler/adapters/terraform_test.go

Length of output: 112342


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- step executor: process and UI output boundaries ---'
rg -n -C 10 --glob '!**/*_test.go' --type go \
  '(type StepExecutor|func \(.*StepExecutor.*\) Execute|exec\.Command|process\.|os\.(Stdout|Stderr)|ui\.[A-Za-z0-9_]+\s*\(|fmt\.(Fprint|Print))' \
  pkg/runner/step pkg/runner

printf '%s\n' '--- store hook execution: terminal and subprocess output boundaries ---'
ast-grep outline pkg/hooks/store_cmd.go --items all
rg -n -C 10 --glob '!**/*_test.go' --type go \
  '(func \(.*StoreCommand.*\) RunE|func .*RunE|exec\.Command|process\.|os\.(Stdout|Stderr)|ui\.[A-Za-z0-9_]+\s*\(|fmt\.(Fprint|Print))' \
  pkg/hooks pkg/store

printf '%s\n' '--- CI log-group write path ---'
sed -n '1,140p' pkg/ci/log_group.go
sed -n '145,230p' pkg/ci/loggroup.go
rg -n -C 8 --glob '!**/*_test.go' --type go \
  '(func .*StartGroup|func .*EndGroup|func .*Group|os\.(Stdout|Stderr)|fmt\.(Fprint|Print)|WriteString)' \
  pkg/ci

printf '%s\n' '--- every ComponentNodeHooks implementation and the legacy fallback ---'
rg -n -C 10 --glob '!**/*_test.go' --type go \
  '(ComponentNodeHooks|BeforeWithWriters|AfterWithWriters|func \(.*\) Before\(|func \(.*\) After\()' \
  . -g '*.go'

printf '%s\n' '--- output and integration tests that cover non-command lifecycle hooks ---'
rg -n -C 8 --type go \
  '(step.*hook|store.*hook|CI.*group|DryRun|dry run|MarkdownMessage|Summary\.Body|ComponentNodeHooksWithOutput)' \
  pkg/hooks pkg/runner pkg/scheduler/adapters cmd

Length of output: 164527


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '%s\n' '--- runner step candidate files ---'
fd -e go . pkg/runner/step | sort
printf '%s\n' '--- store hook candidate files ---'
fd -e go . pkg/hooks | rg '(store|command)' | sort

printf '%s\n' '--- exact runner-step execution and output calls ---'
rg -n -C 16 --glob '!**/*_test.go' --type go \
  'type StepExecutor|func \(.*StepExecutor.*\) Execute|func \(.*StepExecutor.*\) execute|os\.(Stdout|Stderr)|ui\.[A-Za-z0-9_]+\s*\(|exec\.Command|process\.' \
  pkg/runner/step

printf '%s\n' '--- exact store-hook execution and output calls ---'
rg -n -C 16 --glob '!**/*_test.go' --type go \
  'func \(.*StoreCommand.*\) RunE|func \(.*StoreCommand.*\) Run|os\.(Stdout|Stderr)|ui\.[A-Za-z0-9_]+\s*\(|exec\.Command|process\.' \
  pkg/hooks

printf '%s\n' '--- CI group marker implementation ---'
sed -n '1,120p' pkg/ci/log_group.go
sed -n '157,215p' pkg/ci/loggroup.go
rg -n -C 12 --glob '!**/*_test.go' --type go \
  '(StartGroup|EndGroup|StartLogGroup|EndLogGroup|WriteString|os\.Stdout|os\.Stderr)' \
  pkg/ci/internal pkg/ci/providers pkg/ci

Length of output: 150372


@osterman You are right. I should have done this audit before making any blocking comment.

Here is the complete result from the built-in concurrent Terraform paths.

Blocking paths

  1. Structured command-hook output bypasses component writers.
    pkg/hooks/command_engine.go renderTerminal uses ui.Writeln and ui.MarkdownMessage directly.
    This affects scanner summaries, Infracost summaries, and Markdown artifacts.
    The output is neither serialized with Terraform output nor component-prefixed.

  2. kind: step and kind: steps bypass component writers.
    pkg/hooks/step_engine.go invokes pkg/runner/step.StepExecutor.
    Step handlers write directly through ui, data, and terminal APIs.
    This includes ui.ClearLine() in the clear step and spinner use in container steps.
    Shell, script, container, and nested Atmos steps can also render output through their own output-mode writers.
    Therefore, a concurrent lifecycle hook that uses kind: step or kind: steps can still interleave with prefixed Terraform output or emit terminal control output.

These are the two remaining blocking issues for the concurrent terminal-rendering contract.

Non-blocking follow-ups

  1. Concurrent dry-run output is unprefixed.
    TerraformDispatcher.Dispatch calls ui.Successf directly for each dry-run node.
    This output does not contain carriage-return control output.
    It is still inconsistent with component-prefixed concurrent output.

  2. CI log groups use process-wide state.
    pkg/ci/log_group.go uses a global logGroupDepth.
    Concurrent hooks can suppress each other’s CI group markers.
    This affects CI log structure. It is not a terminal rendering corruption path.

  3. Legacy schema.ComponentNodeHooks implementations cannot receive writers.
    The new schema.ComponentNodeHooksWithOutput interface handles the built-in Terraform and Helmfile node hooks.
    A custom implementation that only satisfies the legacy interface can still write directly to the terminal.
    This is an API limitation. It is outside the built-in hook paths audited here.

I found no additional built-in direct terminal-output path in store hooks. command and tfmigrate subprocess streams now use the component writers.

I will use this full-path audit method for this class of review. I will not report another isolated “one remaining blocking gap” unless a code change creates a new output path.

(ᵔᴥᵔ)


✏️ Learnings added
Learnt from: osterman
URL: https://github.com/cloudposse/atmos/pull/2860

Timestamp: 2026-08-04T22:05:21.806Z
Learning: For reviews of concurrent Terraform execution in `pkg/scheduler/adapters/terraform.go`, perform and report one consolidated end-to-end audit of all built-in output boundaries before raising blocking findings. Include Terraform/OpenTofu streams, Atmos transient UI, JIT provisioning, lifecycle hook engines (`command`, `tfmigrate`, `step`, `steps`, and `store`), structured hook summaries, dry-run output, and CI log-group output. Do not report serial “one remaining blocking gap” findings from partial path inspections.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

You are interacting with an AI system.

@zack-is-cool

zack-is-cool commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor Author

kind: step and kind: steps bypass component writers.

Agreed: this is a genuine concurrent-output gap. The contained structured-summary path can be routed through the existing node writer, but step hooks need an execution-scoped output contract propagated through StepExecutor and the relevant handlers/subprocess paths.

A scoped redirect of the global UI writer would not be safe under parallel node execution, because it is process-wide and concurrent hooks could overwrite each other's destination. I recommend handling step-hook output as a dedicated follow-up rather than expanding this PR with a partial workaround. The dry-run, CI-log-group, and legacy-interface items are non-blocking as described.

CodeRabbit (@coderabbitai)

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

@zack-is-cool Agreed. Keep the kind: step and kind: steps output contract out of this PR.

A process-wide UI-writer redirect is unsafe during parallel execution. The follow-up should add execution-scoped writers to StepExecutor, its handlers, and subprocess paths. It should preserve component prefixing, serialization, and carriage-return normalization.

The current PR can treat the structured-summary path as the contained remaining change. The dry-run, CI log-group, and legacy-interface items remain non-blocking.

Would you like me to create a follow-up issue? GitHub Issues and Linear are available.

🐇


✏️ Learnings added
Learnt from: zack-is-cool
URL: https://github.com/cloudposse/atmos/pull/2860

Timestamp: 2026-08-04T22:07:49.897Z
Learning: For concurrent Terraform lifecycle hooks in `pkg/scheduler/adapters/terraform.go`, `kind: step` and `kind: steps` require an execution-scoped output-writer contract propagated through `pkg/runner/step.StepExecutor`, relevant step handlers, and subprocess paths. Do not redirect the global UI writer because it is process-wide and unsafe when parallel nodes execute concurrently. Treat this work as a dedicated follow-up when a partial implementation would not cover all output paths.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

You are interacting with an AI system.

@zack-is-cool

Copy link
Copy Markdown
Contributor Author

CodeRabbit (@coderabbitai) review

@coderabbitai

coderabbitai Bot commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai[bot]
coderabbitai Bot previously approved these changes Aug 4, 2026
@aknysh
Andriy Knysh (aknysh) merged commit 13bce50 into cloudposse:main Aug 5, 2026
95 checks passed
@atmos-pro

atmos-pro Bot commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Tip

Atmos Pro  

No affected stacks workflow was detected for this pull request.
If this is expected, no action is needed.
Learn More. Ask AI.

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown

These changes were released in v1.225.0.

This branch was previously deployed

1 inactive deployment
screengrabs — 220dace0 Deployed Aug 5, 2026 by aknysh via build #1065
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

patch A minor, backward compatible change size/m Medium size PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Concurrent Terraform output can corrupt terminal lines

3 participants