Implement atmos toolchain to install 3rd party tools

[goruha]

Describe the Feature

Atmos should automatically add the packages base-path as a priority search PATH when executing any commands.

Expected Behavior

Packages are automatically installed and available to atmos components, custom commands, and workflows.

When running atmos terraform plan and it depends on opentofu@1.10.3, if not installed it should automatically install it (if it's configured).

When running a custom command, that needs tflint, it should be able to automatically install it.

When running a workflow that needs tfsec, it should automatically install it.

Available Commands:
  add         Add or update a tool and version in .tool-versions
  aliases     List configured tool aliases
  clean       Remove all installed tools by deleting the .tools directory
  completion  Generate the autocompletion script for the specified shell
  exec        Exec a specific version of a tool (replaces current process)
  get         Show all versions configured for a tool, sorted in semver order
  help        Help about any command
  info        Display tool configuration from registry
  install     Install a CLI binary from the registry
  list        List configured tools and their installation status
  path        Emit the complete PATH environment variable for configured tool versions
  remove      Remove a tool or a specific version from .tool-versions
  run         Run a specific version of a tool
  set         Set a specific version for a tool in .tool-versions
  uninstall   Uninstall a CLI binary from the registry
  which       Display the path to an executable

Flags:
  -h, --help                   help for toolchain
      --log-level string       Set log level (debug, info, warn, error)
      --tool-versions string   Path to tool-versions file (default ".tool-versions")
      --tools-config string    Path to tools configuration file (default "tools.yaml")
      --tools-dir string       Directory to store installed tools (default ".tools")

Use Case

• Install versions of opentofu or terraform
• Install helmfile
• Install terraform-docs, tflint, etc.
• Support multiple concurrent versions
• Install any other binaries needed by workflows, custom commands, etc
• Atmos should act as a wrapper for atmos, to install and exec any version of atmos using the use keyword

Describe Ideal Solution

Atmos Commands

# Install all packages
atmos toolchain install

# Update packages 
atmos toolchain update

Atmos Configuration

# atmos.yaml

# Automatically install (if not installed) and use this version of atmos, and exec (replacing PID).
# See `exec` example in `toolchain-experiment`
# THis replaces the current process, it's NOT A SUBSHELL.
# This ensures everyone uses the right version based on the configuration.
use 1.183.1


# Define packages required for a specific type of component
# In this example, we're refering to the "opentofu" alias and 
# specifying the version of 1.10.3
components:
  terraform:
    command: opentofu@1.10.3
  helmfile:
    command: helmfile@1.1.3

The use keyword will install atmos at version 1.183.1, then call syscall.Exec to replace the current process with the atmos at the correct version, if the current version is not already 1.183.1.

Aliases

Aliases map a short name like opentofu to a registry configuration of opentofu/opentofu or terraform to something like hashicorp/terraform

# toolchain subcommand configuration
toolchain:

  
  # Tool name aliases for .tool-versions compatibility
  # Maps common tool names to their registry owner/repo paths
  aliases:
    terraform: hashicorp/terraform
    opentofu: opentofu/opentofu
    helm: helm/helm
    kubectl: kubernetes-sigs/kubectl
    kustomize: kubernetes-sigs/kustomize
    kind: kubernetes-sigs/kind
    krew: kubernetes-sigs/krew
    github-comment: suzuki-shunsuke/github-comment
    tflint: terraform-linters/tflint
    tfsec: aquasecurity/tfsec
    checkov: bridgecrewio/checkov
    terragrunt: gruntwork-io/terragrunt
    packer: hashicorp/packer
    vault: hashicorp/vault
    consul: hashicorp/consul
    nomad: hashicorp/nomad
    waypoint: hashicorp/waypoint
    boundary: hashicorp/boundary
    helmfile: helmfile/helmfile
    atmos: cloudposse/atmos

Tool Versions

We should use the simple .tool-versions convention popularized by asdf

tool1 1.2.3 4.5.7
tool2 2.3.4
tool3 5.6.7

The first semver is the default version, when nothing is specified

This file is consulted when calling install/uninstall/add/remove.

We should use aliases in this file to refer to toolchain tools. This is so we can change the mapping in the future.

Aqua Registry Configuration

We're going to add partial Aqua support for registries and expand the implementation over time. To start, we want to support the essential packages we depend on.

Local Configuration

We also want a local toolchain configuration, when not depending on the Aqua registry.

# toolchain subcommand configuration
toolchain:

  # this is relative to the repo root
  install-dir: .tools/bin

  # Local tools configuration
  # This file is consulted first before the Aqua registry
  # Allows local overrides and custom tool definitions
  

  tools:
    # Atmos configuration supporting both raw and gzipped binaries
    cloudposse/atmos:
      type: github_release
      repo_owner: cloudposse
      repo_name: atmos
      binary_name: atmos
      version_constraints:
        - constraint: ">= 1.0.0"
          asset: atmos_{{trimV .Version}}_{{.OS}}_{{.Arch}}
          format: raw
        - constraint: ">= 1.0.0"
          asset: atmos_{{trimV .Version}}_{{.OS}}_{{.Arch}}.gz
          format: gzip
  
    # Override a registry tool with local settings
    hashicorp/terraform:
      type: http
      url: https://releases.hashicorp.com/terraform/{{trimV .Version}}/terraform_{{trimV .Version}}_{{.OS}}_{{.Arch}}.zip
      format: zip
      binary_name: terraform
  
    # TFLint configuration
    terraform-linters/tflint:
      type: github_release
      repo_owner: terraform-linters
      repo_name: tflint
      binary_name: tflint
  
    # Custom tool not in registry
    my-custom-tool:
      type: github_release
      repo_owner: myorg
      repo_name: my-custom-tool
      asset: my-custom-tool_{{trimV .Version}}_{{.OS}}_{{.Arch}}.tar.gz
      format: tar.gz
      binary_name: my-custom-tool
  
    # Override OpenTofu to use a specific version constraint
    opentofu/opentofu:
      type: github_release
      repo_owner: opentofu
      repo_name: opentofu
      binary_name: tofu
      version_constraints:
        - constraint: ">= 1.10.0"
          asset: tofu_{{trimV .Version}}_{{.OS}}_{{.Arch}}.tar.gz
          format: tar.gz
        - constraint: "< 1.10.0"
          asset: tofu_{{trimV .Version}}_{{.OS}}_{{.Arch}}.zip
          format: zip
  
    helm/helm:
      type: http
      repo_owner: helm
      repo_name: helm
      url: https://get.helm.sh/helm-v{{.Version}}-{{.OS}}-{{.Arch}}.tar.gz
      format: tar.gz
      binary_name: helm

Stack Configuration

# stacks/_defaults.yaml
components:
  terraform:
    vpc:
      command: hashicorp/terraform@v1.11.4

When the command is evaluated, it's first checked against the toolchain, and if it's not installed, it will be automatically installed.

The PATH for exec will be updated to use the directory containing terraform for version 1.11.4

Workflows

workflows:
  tflint:
    description: "Run tflint on a component"
    needs:
    - tflint@1.2.3
    - opentofu@1.10.3
    steps:
      - name: tflint
        type: shell
        command: |
          cd terraform/components
          tflint --recursive

Custom Commands

# Custom CLI commands
commands:
  - name: tfsec
    description: Run tfsec against a specified Terraform component
    arguments:
      - name: component
        description: Name of the component to scan
    needs:
    - tfsec@1.2.3
    steps:
      # Navigate to the Terraform component directory
      - cd components/terraform/{{ .Arguments.component }}
      # Run tfsec scan
      - tfsec .

Alternatives Considered

Using aqua directly. However, the aims of the Aqua Project are different than our aims and Aqua does not expose an SDK or stable interfaces for other CLIs to use Aqua.

https://github.com/suzuki-shunsuke

Summary by CodeRabbit

• New Features
  • Introduces a top-level “toolchain” CLI with subcommands: install, uninstall, list, add, remove, set (interactive), versions, exec, path, info, aliases, clean, which. Supports .tool-versions, local aliases, GitHub/Aqua registry, caching, progress UI, and PATH emission.
• Configuration
  • Adds a toolchain config block (install_path, file_path, tools_dir, tools_config_file) and updates global flags/help output.
• Documentation
  • Adds comprehensive toolchain command docs and help screengrabs; updates CLI help snapshots.
• Tests
  • Extensive unit tests across toolchain features and CLI outputs.
• Chores
  • Promotes dependencies to direct; updates .gitignore.

[mergify]

Warning

This PR exceeds the recommended limit of 1,000 lines.

Large PRs are difficult to review and may be rejected due to their size.

Please verify that this PR does not address multiple issues.
Consider refactoring it into smaller, more focused PRs to facilitate a smoother review process.

[mergify]

Important

Cloud Posse Engineering Team Review Required

This pull request modifies files that require Cloud Posse's review. Please be patient, and a core maintainer will review your changes.

To expedite this process, reach out to us on Slack in the #pr-reviews channel.

[coderabbitai]

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

Adds a new “toolchain” subsystem: CLI commands under cmd/, core implementation under toolchain/ (installer, registry, HTTP/GitHub clients, version-file management, PATH emission, exec/run, list/info UI, uninstall/clean), schema support for toolchain config, docs and screengrabs, tests, and minor project housekeeping (go.mod, .gitignore, snapshots).

Changes

Cohort / File(s)	Summary
CLI wiring (root + toolchain commands) cmd/root.go, cmd/toolchain*.go	Registers new top-level toolchain command and subcommands (add, remove, set, versions, list, install, uninstall, exec, path, info, clean, aliases, which). Propagates Atmos config to toolchain and binds flags/env (incl. GitHub token).
Schema support pkg/schema/schema.go	Adds Toolchain config struct and AtmosConfiguration.Toolchain field with paths for install, versions file, tools dir, and tools config file.
Toolchain core: installer, resolver, types, registry toolchain/installer.go, toolchain/types.go, toolchain/aqua_registry.go, toolchain/local_config.go, toolchain/github.go, toolchain/http_client.go, toolchain/setup.go	Implements installer workflow (resolve → download/cache → extract → place binary), tool/registry data models, local config/aliases, Aqua registry client, GitHub releases client, HTTP client with optional GitHub auth, and toolchain path defaults/getters.
Toolchain features: operations and UX toolchain/install.go, toolchain/uninstall.go, toolchain/exec.go, toolchain/list.go, toolchain/path.go, toolchain/info.go, toolchain/add.go, toolchain/remove.go, toolchain/get.go, toolchain/set.go, toolchain/clean.go, toolchain/progress.go, toolchain/tui.go	Adds install/uninstall flows, exec (process replace), listing with table UI, PATH emission (plain/export/JSON), info output (table/YAML), add/remove/set/versions commands, clean of tools and caches, and small TTY helpers/TUI components.
Toolchain tests (core + features) toolchain/*_test.go	Comprehensive unit tests for installer, registry, HTTP/GitHub clients, local config, versions parsing/manipulation, add/remove/set/get/list/path/which/exec/uninstall, and helpers/mocks.
Config/data for toolchain toolchain/tools.yaml, toolchain/test-package.yaml, toolchain/README.md, toolchain/Makefile	Adds local aliases/tools config, a sample package YAML, README documenting architecture/usage, and a Makefile for the toolchain module.
Docs: CLI pages website/docs/cli/commands/toolchain/*	Adds toolchain command docs (usage, flags, examples) and a category entry.
Docs: Screengrabs website/src/components/Screengrabs/atmos-toolchain-*.html, website/src/components/Screengrabs/atmos--help.html, website/src/components/Screengrabs/demo-stacks/*	Adds static HTML screengrabs for toolchain help/commands and updates Atmos/help and demo-stack visuals/messages.
Snapshots and tests outside toolchain tests/snapshots/TestCLICommands_atmos_--help.stdout.golden, tests/snapshots/TestCLICommands_atmos_describe_config.stdout.golden, tests/snapshots/TestCLICommands_atmos_non-existent.stderr.golden, internal/exec/atlantis_utils_test.go	Updates CLI help/config snapshots to include toolchain; adds Atlantis naming tests.
Project housekeeping .gitignore, go.mod	Ignores node_modules; promotes semver and mimetype to direct deps.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor User
  participant CLI as atmos toolchain exec
  participant TC as toolchain.RunExecCommand
  participant Inst as Installer
  participant FS as Filesystem
  participant OS as OS (exec)

  User->>CLI: atmos toolchain exec terraform@1.11.4 --version
  CLI->>TC: RunExecCommand(args)
  TC->>Inst: GetResolver().Resolve("terraform")
  alt binary exists
    TC->>Inst: FindBinaryPath(owner, repo, version)
  else not installed
    TC->>Inst: (ensure) Install(owner, repo, version)
    Inst->>FS: download/cache/extract/install
    FS-->>Inst: path
    TC->>Inst: FindBinaryPath(...)
  end
  TC->>OS: exec(binaryPath, args, env)
  OS-->>User: replaced process

Loading

sequenceDiagram
  autonumber
  actor User
  participant CLI as atmos toolchain install
  participant Flow as RunInstall
  participant Vers as .tool-versions
  participant Inst as Installer
  participant Reg as Registry (local/aqua)
  participant FS as Filesystem

  User->>CLI: atmos toolchain install
  CLI->>Flow: RunInstall("", setDefault=false, reinstall=false)
  Flow->>Vers: LoadToolVersions(file)
  loop each tool@versions
    Flow->>Reg: Resolve owner/repo
    alt already installed and !reinstall
      Flow->>Flow: skip
    else install
      Flow->>Inst: Install(owner, repo, version)
      Inst->>FS: download/cache/extract
      FS-->>Inst: binary path
    end
  end
  Flow-->>User: summary (installed/failed/skipped)

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

Possibly related PRs

• refactor loading and checking of atmos config #869 — Also updates cmd/root.go around Atmos config initialization; related to this PR’s addition of toolchain.SetAtmosConfig and CLI wiring.
• Improve Atmos help #959 — Modifies Execute flow in cmd/root.go to initialize subsystems and handle flags; overlaps with this PR’s root wiring for toolchain.

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch tools-experiment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

❌ Patch coverage is 57.10211% with 1504 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.11%. Comparing base (f2781b5) to head (859c1f1).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
toolchain/installer.go	52.91%	289 Missing and 50 partials ⚠️
toolchain/uninstall.go	41.42%	164 Missing and 17 partials ⚠️
toolchain/aqua_registry.go	52.95%	145 Missing and 14 partials ⚠️
toolchain/set.go	57.14%	148 Missing and 2 partials ⚠️
toolchain/tool_versions.go	43.29%	139 Missing and 9 partials ⚠️
toolchain/install.go	49.76%	86 Missing and 19 partials ⚠️
toolchain/list.go	78.54%	56 Missing and 15 partials ⚠️
toolchain/get.go	71.64%	30 Missing and 8 partials ⚠️
toolchain/info.go	49.33%	36 Missing and 2 partials ⚠️
toolchain/tui.go	0.00%	37 Missing ⚠️
... and 22 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1386      +/-   ##
==========================================
+ Coverage   55.94%   56.11%   +0.17%     
==========================================
  Files         274      311      +37     
  Lines       28936    32441    +3505     
==========================================
+ Hits        16188    18204    +2016     
- Misses      10956    12271    +1315     
- Partials     1792     1966     +174     

Flag	Coverage Δ
unittests	56.11% <57.10%> (+0.17%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mergify]

💥 This pull request now has conflicts. Could you fix it @goruha? 🙏

[osterman]

Let's close this and have you (@samtholiya) open the pull request.

[mergify]

⚠️ The sha of the head commit of this PR conflicts with #1466. Mergify cannot evaluate rules on this PR. ⚠️