Add short test mode to skip long-running tests

[Erik Osterman (Cloud Posse) (osterman)]

what

• Add support for Go's -short flag to skip long-running tests (>2 seconds)
• Enable faster development feedback loop while preserving comprehensive CI testing
• Add SkipIfShort() helper function for Go tests
• Add short field to YAML test case schema (defaults to true)
• Mark 13 Go tests and 9 YAML tests as long-running
• Add make test-short and make test-short-cover Makefile targets
• Fix AWS profile precondition failures in 2 auth tests

why

• Developers need faster test feedback during development
• Full test suite takes 3+ minutes, making rapid iteration slow
• Many tests require network I/O, Git operations, or binary compilation (>2s each)
• CI should run all tests, but local development benefits from quick tests
• Follows Go's standard -short flag convention for test skipping

Performance Impact

Before: Full test suite only (~3+ minutes)

After:

• Quick mode (make test-short): ~2m30s (skips 30+ seconds of slow tests)
• Full mode (make testacc): ~3m+ (runs everything including slow tests)

Changes by Category

Go Tests Marked Long (13 tests)

• Git operations (3 tests, ~60s saved):
  • TestDescribeAffectedWithTargetRefClone (36s - Git cloning)
  • TestExecuteAtlantisGenerateRepoConfigAffectedOnly (21s - Git ops)
  • TestExecuteTerraformAffectedWithDependents (26s - Git + Terraform)
• Network I/O (3 tests, ~10s saved):
  • TestVendorComponentPullCommand (6s)
  • TestVendorPullFullWorkflow (network + OCI)
  • TestVendorPullBasicExecution (4s)
• Binary compilation (7 tests in testhelpers, ~100s saved):
  • All atmos binary build tests (15-20s each)
• AWS SDK (2 tests, now with proper preconditions):
  • TestAssumeRoleIdentity_newSTSClient_RegionFallbackAndPersist
  • TestPermissionSetIdentity_newSSOClient_Success

YAML Tests Marked Long (9 tests)

All vendor pull tests requiring network I/O:

• tests/test-cases/vendor-test.yaml (2 tests)
• tests/test-cases/demo-vendoring.yaml (2 tests)
• tests/test-cases/demo-globs.yaml (1 test)
• tests/test-cases/vendoring-ssh-dryrun.yaml (3 tests)

Infrastructure

• New helper: tests.SkipIfShort(t) in tests/preconditions.go
• Schema update: Added short field to tests/test-cases/schema.json
• CLI framework: Modified tests/cli_test.go to check short mode before running tests
• Makefile targets: test-short and test-short-cover
• Documentation: Updated CLAUDE.md and tests/README.md

Testing

# Quick tests pass in ~2m30s
make test-short

# All tests still pass (full suite)
make testacc

# Verify long tests are skipped
go test -short -v ./internal/exec | grep SKIP

# Verify long tests run without -short
go test -v ./internal/exec | grep "TestExecuteTerraformAffectedWithDependents"

Usage

# Development: quick feedback
make test-short
go test -short ./...

# CI/comprehensive testing
make testacc
go test ./...

# With coverage
make test-short-cover
go test -short -cover ./...

references

• Follows Go testing conventions: https://pkg.go.dev/testing#Short
• Related to test precondition system in tests/preconditions.go

🤖 Generated with Claude Code

[mergify]

Important

Cloud Posse Engineering Team Review Required

This pull request modifies files that require Cloud Posse's review. Please be patient, and a core maintainer will review your changes.

To expedite this process, reach out to us on Slack in the #pr-reviews channel.

[codecov]

Codecov Report

❌ Patch coverage is 50.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.52%. Comparing base (41200b0) to head (d18c14e).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
tests/preconditions.go	50.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1605      +/-   ##
==========================================
+ Coverage   63.49%   63.52%   +0.02%     
==========================================
  Files         323      323              
  Lines       37061    37067       +6     
==========================================
+ Hits        23533    23547      +14     
+ Misses      11580    11568      -12     
- Partials     1948     1952       +4     

Flag	Coverage Δ
unittests	63.52% <50.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
tests/preconditions.go	80.32% <50.00%> (-0.75%)	⬇️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

These changes were released in v1.194.0.