Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
4554d31
feat(workflows): add test step with buffered failure reporting
osterman Sep 11, 2026
ebad56a
chore(build): display binary build output in a viewport
osterman Sep 11, 2026
3dc5b4e
chore(build): limit build viewport to four lines
osterman Sep 11, 2026
aedd33f
feat(build): add no-cache flag for forced recompilation
osterman Sep 11, 2026
2a8b6d6
fix(cli): display help when invoked without a command
osterman Sep 11, 2026
ce87ee3
fix(ui): render live viewports and refine test reports
osterman Sep 11, 2026
c8999bf
test: align CI assertions with test styling and root help
osterman Sep 11, 2026
4b6a6b5
feat: show total elapsed time in test summaries
osterman Sep 11, 2026
8957e0f
test: cover test runner failures and validation paths
osterman Sep 11, 2026
7a94109
fix: address test runner review feedback
osterman Sep 12, 2026
16eb4fa
fix: preserve test progress colors in recordings
osterman Sep 12, 2026
5ac07c5
Merge remote-tracking branch 'origin/main' into osterman/test-runner-…
osterman Sep 12, 2026
f68fa69
fix: align cast terminal rails and improve line spacing
osterman Sep 12, 2026
17c95f0
test: make cache contention check independent of CI timing
osterman Sep 12, 2026
e6fa425
docs: add Atmos test authoring skill and execution patterns
osterman Sep 12, 2026
3eaef84
test: isolate viewport TTY state and remove hook provider downloads
osterman Sep 12, 2026
6cfb3f4
test: restore viewport env binding after Viper resets
osterman Sep 12, 2026
ef38b63
docs: show test recording preview on changelog timeline
osterman Sep 13, 2026
a9fa14f
fix: add cast downloads to test documentation
osterman Sep 13, 2026
0e0548c
fix: use themed colors for test spinners
osterman Sep 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .atmos.d/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,19 +10,31 @@ commands:
value: "{{ .Flags.target }}"
- key: ATMOS_BUILD_VERSION
value: "{{ .Flags.version }}"
- key: ATMOS_BUILD_NO_CACHE
value: '{{ index .Flags "no-cache" }}'
flags:
- name: target
description: "Build target: default, linux, windows, macos, or macos-intel"
default: default
- name: version
description: Version string to embed in the binary
default: test
- &build_no_cache_flag
name: no-cache
type: bool
description: Force rebuilding all Go packages, including dependencies
default: false
steps:
- type: toast
level: info
content: Building Atmos...
- &build_binary_step
name: Compiling
type: shell
output: viewport
viewport:
height: 4
padding: 2
command: |
set -eu
go tool mage build:binary "${ATMOS_BUILD_TARGET:-default}" "${ATMOS_BUILD_VERSION:-test}"
Expand Down Expand Up @@ -61,6 +73,7 @@ commands:
- name: version
description: Version string to embed in the binary
default: test
- *build_no_cache_flag
steps:
- type: toast
level: info
Expand Down
6 changes: 3 additions & 3 deletions .claude/skills/changelog/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,9 +135,9 @@ import CastEmbed from '@site/src/components/CastEmbed'
- Always carry the `chrome controls scrubber` flags.
- Multiple `<CastEmbed>` tags are fine in one post if there are multiple relevant recordings.
- `CastEmbed` wraps `CastPlayer` and adds Download (rendered GIF/MP4/SVG/WEBM via Atmos Pro) and Share controls,
on by default against `cloudposse/atmos` @ `main`. If the `.cast` file isn't committed to `main` yet (e.g. it
ships in the same PR as the post), pass `download={false}` and/or `share={false}` to suppress the controls
until it lands.
on by default against `cloudposse/atmos` at the site build's Git commit (`GITHUB_SHA` in CI,
`main` for local builds). This lets PR previews download recordings introduced by the same PR.
Use `gitRef` to override the source revision explicitly; do not hide controls just because a cast is unmerged.
- Follow it with a plain link to the full example when one exists: `[View the full example](/examples/<name>)`.
- Don't use `EmbedExample` in blog posts — that component's README/file-listing duplicates content the post's
own prose already covers; it's for docs pages that need the "browse the full example" callout instead.
Expand Down
1 change: 1 addition & 0 deletions agent-skills/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,7 @@ When a task involves Atmos, activate the matching skill for detailed guidance.
| ansible playbook execution, variable passing, inventory management, configuration management | `atmos-ansible` | `agent-skills/skills/atmos-ansible/SKILL.md` |
| Terminal asciicast demos for community-facing docs, examples, and training materials | `atmos-asciicast` | `agent-skills/skills/atmos-asciicast/SKILL.md` |
| Shared step DSL: step types, env, output, working_directory, retry, script, workdir, cast, hook `with:` payloads | `atmos-steps` | `agent-skills/skills/atmos-steps/SKILL.md` |
| Smoke tests and integration tests: test groups, HTTP assertions, script/interpreter, parallel, matrix, post-deployment checks | `atmos-tests` | `agent-skills/skills/atmos-tests/SKILL.md` |
| Multi-step workflows, native step types, `when:` CEL conditions, `require`/`assert` preconditions, background steps, output/UI steps, cross-component orchestration | `atmos-workflows` | `agent-skills/skills/atmos-workflows/SKILL.md` |
| Cast recording/rendering: cast play/render, format inference, `--cast` flag, `type: cast`/`type: simulate` steps | `atmos-cast` | `agent-skills/skills/atmos-cast/SKILL.md` |
| Custom CLI commands in atmos.yaml, arguments, flags, native steps, `when:` conditions, custom component types, env vars, subcommands | `atmos-custom-commands` | `agent-skills/skills/atmos-custom-commands/SKILL.md` |
Expand Down
1 change: 1 addition & 0 deletions agent-skills/skills/atmos-custom-commands/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ commands:
| Complete command schema and examples | [references/command-syntax.md](references/command-syntax.md) |
| Reusable multi-step orchestration | `atmos-workflows` |
| Shared step fields and step types | `atmos-steps` |
| Smoke tests, integration tests, and test groups | `atmos-tests` |
| Tool versions and PATH behavior | `atmos-toolchain` |
| Auth providers, identities, assume role/root, OIDC | `atmos-auth` |
| Components and component inheritance | `atmos-components` |
Expand Down
4 changes: 3 additions & 1 deletion agent-skills/skills/atmos-hooks/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ toolchain-aware automation around Terraform, Helm, Kubernetes, and other compone
|---|---|
| Store output hooks | [atmos-stores](../atmos-stores/SKILL.md) |
| Shared step fields and `kind: step` payloads | [atmos-steps](../atmos-steps/SKILL.md) |
| Post-deployment smoke tests and integration checks | [atmos-tests](../atmos-tests/SKILL.md) |
| Git hooks and GitOps repositories | [atmos-git](../atmos-git/SKILL.md) |
| Tool installation for hook commands | [atmos-toolchain](../atmos-toolchain/SKILL.md) |
| CI summaries and Atmos Pro upload | [atmos-ci](../atmos-ci/SKILL.md) and [atmos-pro](../atmos-pro/SKILL.md) |
Expand Down Expand Up @@ -123,7 +124,8 @@ recordings use, instead of one of the named kinds above:
configure it with `with:`, exactly like a workflow step.
- `kind: steps` runs an ordered list of registered step types, provided as a YAML list under `with:`.

Both run strictly in order -- there is no concurrent execution within a step-backed hook.
Hook step lists run in order. A `type: test` group can contain `parallel` or
`matrix` checks; see [atmos-tests](../atmos-tests/SKILL.md).

The hook envelope owns `events`, `when`, `env`, `retry`, and `on_failure`; `with:` is
decoded and validated as the step's own configuration. `kind: step` supplies the one
Expand Down
7 changes: 5 additions & 2 deletions agent-skills/skills/atmos-steps/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ Important shared fields:
- `script` and `interpreter`: Inline script body and runtime for `type: script`.
- `working_directory`: Directory for the subprocess or script.
- `env`: Map of environment variables layered onto the step.
- `output`: Output mode: `raw`, `log`, `viewport`, or `none`.
- `output`: Output mode: `raw`, `log`, `viewport`, or `none`; `test` groups instead use `failures` or `all`.
- `retry`: Retry policy around the whole step.
- `identity`: Atmos identity used when the step runs.
- `when`: Declarative condition for whether the step runs.
Expand All @@ -95,7 +95,7 @@ as the canonical reference. Current canonical step types include:

- Command and integration: `atmos`, `shell`, `script`, `exec`, `container`,
`emulator`, `http`, `archive`, `require`, `workdir`, `cast`, `store`.
- Orchestration: `parallel`, `matrix`, `wait`, `wait-all`, `cancel`.
- Orchestration: `test`, `parallel`, `matrix`, `wait`, `wait-all`, `cancel`.
- Interactive: `input`, `confirm`, `choose`, `filter`, `file`, `write`.
- UI and output: `toast`, `markdown`, `spin`, `table`, `pager`, `format`,
`join`, `style`, `log`, `junit`, `hint`, `alert`, `say`, `title`, `clear`,
Expand All @@ -107,6 +107,9 @@ document and configure the canonical names unless compatibility requires an alia
If code and docs disagree, inspect the registered step handlers under
`pkg/runner/step/` and schema constants in `pkg/schema/task.go`.

For smoke tests and integration checks, use [atmos-tests](../atmos-tests/SKILL.md)
for test groups, assertions, parallel dependencies, and matrix cases.

## Environment

Prefer map syntax:
Expand Down
121 changes: 121 additions & 0 deletions agent-skills/skills/atmos-tests/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
---
name: atmos-tests
description: "Author Atmos smoke tests, post-deployment stack checks, and integration tests using type: test, HTTP assertions, shell or script/interpreter checks, require gates, parallel dependencies, matrix cases, and lifecycle hooks. Use for test suites in Atmos configuration, not Go unit tests of Atmos itself."
metadata:
copyright: Copyright Cloud Posse, LLC 2026
version: "1.0.0"
category: ci-automation
---

# Atmos Tests

Use `type: test` to run smoke tests that validate deployed stacks or other
integration tests. It groups existing typed steps into a test report: passing
logs stay hidden, failures reveal their buffered output, and independent checks
continue by default.

Read [patterns](references/patterns.md) for complete custom-command, HTTP,
parallel/dependency, matrix, script/interpreter, and post-apply hook examples.
Use [atmos-steps](../atmos-steps/SKILL.md) for shared step fields and the relevant
surface skill: [custom commands](../atmos-custom-commands/SKILL.md),
[workflows](../atmos-workflows/SKILL.md), or [hooks](../atmos-hooks/SKILL.md).

## Authoring Process

1. Inspect the project's commands, workflows, hooks, and existing test scripts.
Discover actual stack/component names and deployment outputs before choosing
targets. Do not invent endpoints or hard-code credentials.
2. Prefer a custom command for a directly invoked suite with its own arguments
or flags. Use a workflow for an existing orchestration sequence, or a lifecycle
hook to run checks after deployment. There is no built-in standalone `atmos test`
command: defining a custom command named `test` creates that invocation.
3. Choose the smallest step type that expresses each assertion. A command that
merely prints a response is not an assertion; it must fail when expectations
are unmet. Name cases descriptively and use `title` for readable display labels.
4. Keep direct children sequential. Use a sibling `parallel` or `matrix` group
when cases should run concurrently. Set a concurrency limit suitable for the
service and isolate mutable fixtures for each concurrent case.
5. Keep default failure-only output and continuation unless the test contract
requires otherwise. Add bounded retries for eventual consistency; do not use
retries to conceal a reproducible assertion failure.
6. Verify a passing case and an intentionally failing case against local fixtures
or an appropriate test environment. Check the exit status, failure details,
continuation, and expanded leaf totals. Restore the intended expectations.

## Choose a Test Step

| Need | Step and pattern |
|---|---|
| HTTP status, health endpoint, or response text | `http` with `expect.status` and optionally `expect.response`; prefer this over `curl` and shell parsing. |
| JSON structure, numerical comparisons, or multi-part assertions | `script` with explicit `interpreter` and `script`; consume an HTTP step's response through its result value or use a language client. |
| Existing test script or external CLI | `shell` with `command`; ensure assertion failures propagate as a nonzero exit. Prefer checked-in scripts for substantial logic. |
| Required tools, files, or directories | `require` with `tools`, `files`, or `dirs`; useful as a precondition, not proof that a service is healthy. |
| Atmos validation or inspection with meaningful exit status | `atmos` with a command such as `validate stacks`; inspect-only commands need a separate assertion on their output. |
| An isolated test runner image | Foreground `container` with a finite test command; consult its step documentation for image/runtime fields. Do not detach it or request a TTY. |
| Independent named checks | `parallel` with `max_concurrency`; use child `needs` for prerequisites and result consumption. |
| The same checks across regions, endpoints, runtimes, or other axes | `matrix` with named axis lists and `max_concurrency`; use `{{ .matrix.<axis> }}` in child fields. |

`script` requires both `interpreter` and `script`; do not put `command` on it.
Python, Node.js, and other interpreters are patterns, not new step types. Declare
needed runtimes using the owning command/workflow's `dependencies.tools` and the
project's toolchain conventions. `require` verifies availability; it never installs.

## Execution and Failure Rules

- Direct children run in declaration order. `test` does not accept
`max_concurrency`; put it on `parallel` or `matrix`.
- `needs` is supported on children of `parallel`/`matrix`, not direct children of
`test`. Use it whenever a check consumes another concurrent check's result.
A failed dependency skips its dependent checks.
- Parallel and matrix groups can be siblings under `test`, but cannot contain
another parallel/matrix group. Nested `test` groups are unsupported.
- `fail.mode: wait_all` is the default: independent tests continue, then unhandled
failures fail the group. Unset `when` permits continuation after a failure;
explicit `when: success` would instead gate a check on success.
- `fail.mode: fail_fast` cancels remaining work; `fail.max_failures` configures the
existing failure threshold. `best_effort` records failures but lets the group
succeed. Use these deliberately, especially for deployment gates.
- Preserve explicit child `continue` and nested group failure policies. For
example, `continue: always` tolerates a leaf failure, but that case still appears
red in the report. Tolerated failures are not passing assertions.
- `retry` retries a leaf as one case; the final attempt determines its outcome.
Use explicit expectations and bounded attempts for readiness checks.
- Use supported noninteractive registered handlers. Interactive prompts, terminal
handoff, process replacement (`exec`, `exit`), background/async work, recording
or emulator sessions, and `wait`/`wait-all`/`cancel` steps cannot run inside a test.
Start services and prepare fixtures outside the group.

## Output and Results

Set `output: failures` (the default) or `output: all` on the test group. These are
scalar test-specific choices, not `raw`, `none`, `log`, or `viewport`. Both buffer
per leaf; `all` also reveals successful logs at completion. Do not redirect leaf
output to `/dev/null`: it removes the evidence needed when a test fails.

The terminal report uses a hierarchical tree and progress bar. CI/non-TTY output
is static. Counts cover expanded leaves once, excluding group nodes; retries do
not add cases. The summary includes passed, failed, skipped, canceled, and elapsed
time. Group result metadata exposes `total`, `passed`, `failed`, `skipped`, and
`canceled`.

Use scoped environment variables and step result values rather than shared files
or process-global state to pass data. Matrix cases must not overwrite the same
fixture paths. Keep secrets in existing Atmos auth/secret mechanisms; masking
still applies to captured output, but do not deliberately print credentials.

Place optional success messages after the test group with `when: success` and
content such as `Tests passed`. Message-only steps inside the group would count
as cases without validating anything.

## Canonical References

Consult the project docs for exact fields before extending a pattern:

- `website/docs/workflows/workflows/workflow/steps/type/test.mdx`
- The neighboring `http.mdx`, `script.mdx`, `shell.mdx`, `require.mdx`,
`atmos.mdx`, `container.mdx`, `parallel.mdx`, and `matrix.mdx` files.
- `website/docs/workflows/workflows/workflow/steps/continue.mdx` and `retry.mdx`.
- `examples/tests/atmos.yaml` for runnable local examples.

If behavior is unclear, inspect `pkg/schema/test_step.go` and the registered
handlers under `pkg/runner/step/` rather than inventing test-only assertion fields.
Loading
Loading