Implement `atmos toolchain` to install 3rd party tools

[osterman]

Describe the Feature

Atmos should automatically add the packages base-path as a priority search PATH when executing any commands.

Expected Behavior

Packages are automatically installed and available to atmos components, custom commands, and workflows.

When running atmos terraform plan and it depends on opentofu@1.10.3, if not installed it should automatically install it (if it's configured).

When running a custom command, that needs tflint, it should be able to automatically install it.

When running a workflow that needs tfsec, it should automatically install it.

Available Commands:
  add         Add or update a tool and version in .tool-versions
  aliases     List configured tool aliases
  clean       Remove all installed tools by deleting the .tools directory
  completion  Generate the autocompletion script for the specified shell
  exec        Exec a specific version of a tool (replaces current process)
  get         Show all versions configured for a tool, sorted in semver order
  help        Help about any command
  info        Display tool configuration from registry
  install     Install a CLI binary from the registry
  list        List configured tools and their installation status
  path        Emit the complete PATH environment variable for configured tool versions
  remove      Remove a tool or a specific version from .tool-versions
  run         Run a specific version of a tool
  set         Set a specific version for a tool in .tool-versions
  uninstall   Uninstall a CLI binary from the registry
  which       Display the path to an executable

Flags:
  -h, --help                   help for toolchain
      --log-level string       Set log level (debug, info, warn, error)
      --tool-versions string   Path to tool-versions file (default ".tool-versions")
      --tools-config string    Path to tools configuration file (default "tools.yaml")
      --tools-dir string       Directory to store installed tools (default ".tools")

Use Case

• Install versions of opentofu or terraform
• Install helmfile
• Install terraform-docs, tflint, etc.
• Support multiple concurrent versions
• Install any other binaries needed by workflows, custom commands, etc
• Atmos should act as a wrapper for atmos, to install and exec any version of atmos using the use keyword

Describe Ideal Solution

Atmos Commands

# Install all packages
atmos toolchain install

# Update packages 
atmos toolchain update

Atmos Configuration

# atmos.yaml

# Automatically install (if not installed) and use this version of atmos, and exec (replacing PID).
# See `exec` example in `toolchain-experiment`
# THis replaces the current process, it's NOT A SUBSHELL.
# This ensures everyone uses the right version based on the configuration.
use 1.183.1


# Define packages required for a specific type of component
# In this example, we're refering to the "opentofu" alias and 
# specifying the version of 1.10.3
components:
  terraform:
    command: opentofu@1.10.3
  helmfile:
    command: helmfile@1.1.3

The use keyword will install atmos at version 1.183.1, then call syscall.Exec to replace the current process with the atmos at the correct version, if the current version is not already 1.183.1.

Aliases

Aliases map a short name like opentofu to a registry configuration of opentofu/opentofu or terraform to something like hashicorp/terraform

# toolchain subcommand configuration
toolchain:

  
  # Tool name aliases for .tool-versions compatibility
  # Maps common tool names to their registry owner/repo paths
  aliases:
    terraform: hashicorp/terraform
    opentofu: opentofu/opentofu
    helm: helm/helm
    kubectl: kubernetes-sigs/kubectl
    kustomize: kubernetes-sigs/kustomize
    kind: kubernetes-sigs/kind
    krew: kubernetes-sigs/krew
    github-comment: suzuki-shunsuke/github-comment
    tflint: terraform-linters/tflint
    tfsec: aquasecurity/tfsec
    checkov: bridgecrewio/checkov
    terragrunt: gruntwork-io/terragrunt
    packer: hashicorp/packer
    vault: hashicorp/vault
    consul: hashicorp/consul
    nomad: hashicorp/nomad
    waypoint: hashicorp/waypoint
    boundary: hashicorp/boundary
    helmfile: helmfile/helmfile
    atmos: cloudposse/atmos

Tool Versions

We should use the simple .tool-versions convention popularized by asdf

tool1 1.2.3 4.5.7
tool2 2.3.4
tool3 5.6.7

The first semver is the default version, when nothing is specified

This file is consulted when calling install/uninstall/add/remove.

We should use aliases in this file to refer to toolchain tools. This is so we can change the mapping in the future.

Aqua Registry Configuration

We're going to add partial Aqua support for registries and expand the implementation over time. To start, we want to support the essential packages we depend on.

Local Configuration

We also want a local toolchain configuration, when not depending on the Aqua registry.

# toolchain subcommand configuration
toolchain:

  # this is relative to the repo root
  install-dir: .tools/bin

  # Local tools configuration
  # This file is consulted first before the Aqua registry
  # Allows local overrides and custom tool definitions
  

  tools:
    # Atmos configuration supporting both raw and gzipped binaries
    cloudposse/atmos:
      type: github_release
      repo_owner: cloudposse
      repo_name: atmos
      binary_name: atmos
      version_constraints:
        - constraint: ">= 1.0.0"
          asset: atmos_{{trimV .Version}}_{{.OS}}_{{.Arch}}
          format: raw
        - constraint: ">= 1.0.0"
          asset: atmos_{{trimV .Version}}_{{.OS}}_{{.Arch}}.gz
          format: gzip
  
    # Override a registry tool with local settings
    hashicorp/terraform:
      type: http
      url: https://releases.hashicorp.com/terraform/{{trimV .Version}}/terraform_{{trimV .Version}}_{{.OS}}_{{.Arch}}.zip
      format: zip
      binary_name: terraform
  
    # TFLint configuration
    terraform-linters/tflint:
      type: github_release
      repo_owner: terraform-linters
      repo_name: tflint
      binary_name: tflint
  
    # Custom tool not in registry
    my-custom-tool:
      type: github_release
      repo_owner: myorg
      repo_name: my-custom-tool
      asset: my-custom-tool_{{trimV .Version}}_{{.OS}}_{{.Arch}}.tar.gz
      format: tar.gz
      binary_name: my-custom-tool
  
    # Override OpenTofu to use a specific version constraint
    opentofu/opentofu:
      type: github_release
      repo_owner: opentofu
      repo_name: opentofu
      binary_name: tofu
      version_constraints:
        - constraint: ">= 1.10.0"
          asset: tofu_{{trimV .Version}}_{{.OS}}_{{.Arch}}.tar.gz
          format: tar.gz
        - constraint: "< 1.10.0"
          asset: tofu_{{trimV .Version}}_{{.OS}}_{{.Arch}}.zip
          format: zip
  
    helm/helm:
      type: http
      repo_owner: helm
      repo_name: helm
      url: https://get.helm.sh/helm-v{{.Version}}-{{.OS}}-{{.Arch}}.tar.gz
      format: tar.gz
      binary_name: helm

Stack Configuration

# stacks/_defaults.yaml
components:
  terraform:
    vpc:
      command: hashicorp/terraform@v1.11.4

When the command is evaluated, it's first checked against the toolchain, and if it's not installed, it will be automatically installed.

The PATH for exec will be updated to use the directory containing terraform for version 1.11.4

Workflows

workflows:
  tflint:
    description: "Run tflint on a component"
    needs:
    - tflint@1.2.3
    - opentofu@1.10.3
    steps:
      - name: tflint
        type: shell
        command: |
          cd terraform/components
          tflint --recursive

Custom Commands

# Custom CLI commands
commands:
  - name: tfsec
    description: Run tfsec against a specified Terraform component
    arguments:
      - name: component
        description: Name of the component to scan
    needs:
    - tfsec@1.2.3
    steps:
      # Navigate to the Terraform component directory
      - cd components/terraform/{{ .Arguments.component }}
      # Run tfsec scan
      - tfsec .

Alternatives Considered

Using aqua directly. However, the aims of the Aqua Project are different than our aims and Aqua does not expose an SDK or stable interfaces for other CLIs to use Aqua.

https://github.com/suzuki-shunsuke

[suzuki-shunsuke]

There are several options:

1. Use a version manager like aqua
2. Run a version manager like aqua internally
3. Call a version manager like aqua as a Go Module
4. Develop the feature from scratch

The option 1 is simplest and easiest.
We don't need to develop anything.
We only need to encourage users to use a version manager.
But the drawback is that users need to install tools themselves.
By enabling atmos to manage tools, users don't need to install tools themselves.

The option 2 is not bad.
I think we should consider this option first.
I'm the author of aqua, which is a CLI version manager, so I use aqua here.
atmos can generate configuration files of aqua such as aqua.yaml, registry.yaml, aqua-checksums.json, and aqua-policy.yaml internally.
And atmos can execute aqua install internally to install tools.
By adding $(aqua root-dir)/bin to $PATH internally, users don't need to be aware of aqua.
aqua also provides a update command.
aqua provides Renovate config presets. We can utilize them or develop similar presets for Atmos.

This is an advanced topic, but one of problems is checksum verification in CI.
Checksum verification is a feature of aqua.
This feature is very important for security.
aqua recommends managing aqua-checksums.json by Git, otherwise checksums can be tampered.
If atmos hides aqua and doesn't manage aqua-checksums.json by Git, the checksum verification doesn't work enough.
But I'm aware that managing aqua-checksums.json by Git is a bit bothersome because we need to update checksums when we update tools.
Especially, when we update tools by Renovate, we also need to update checksums.
aqua provides a GitHub Action to update checksums automatically, but this requires a GitHub App.

The option 3 is good, but I guess most version managers are not Go Modules but CLI.
aqua is written in Go, and it doesn't use Go's internal packages, which means all packages are public.
But aqua doesn't intend that other tools depend on aqua's packages.
API isn't third party friendly, and API can be changed drastically, so I don't recommend using aqua as a library.

I don't recommend the option 4.
We should utilize existing tools rather than re-inventing the wheel.
We need to maintain something like aqua's registries ourselves.
aqua provides various features.
If other options isn't acceptable, we need to consider this option.

What do you think?

[osterman]

@suzuki-shunsuke

1. Use a version manager like aqua

My main concern is this: it defeats the purpose of the desired functionality, which is to avoid the need to manually install additional tools—including the tooling required to do so (besides Atmos itself). The goal is for everything to "just work" out of the box. I feel so strongly about this that I’d rather hold off until we find a solution that aligns with this principle.

2. Run a version manager like aqua internally

Same response as (1).

3. Call a version manager like aqua as a Go Module

Thank you for clarifying this point!

This was definitely my preferred approach. I was hoping this approach could work, as I noticed that the packages were not strictly internal.

However, I appreciate your warning about the unstable interface and its potential for changes without notice. This raises concerns and gives me pause. Let's see if there's a middle ground (see option 5).

4. Develop the feature from scratch

I would prefer to rule this out as well.

---

I’d like to propose two additional options based on your feedback.

Option 5: Call Aqua (Preferred)

We have an upcoming PR that introduces support for editorconfig.

• Add editorconfig validation #896

What’s notable about this implementation is that while many editorconfig packages are internal, we were able to call their public methods, achieving a near-native integration within Atmos.

What if we did something similar with Aqua?

In this model, if Aqua could expose a few stable APIs that provide command-line-like functionality from within Go, Atmos could call those directly. This way there's not a lot of work to maintain in Aqua, while providing other tooling the ability to leverage aqua. While this may sacrifice full flexibility, it offers the advantage of maintaining a native user experience within Atmos.

As part of this if we could programmatically pass a string containing aqua configuration, we can then work very well with parts of aqua. We could generate that configuration inside of atmos, so the package manafement can feel more integrated with things like stacks, workflows and custom commands.

Option 6: Go-based Aqua Bootstrapper (Less Preferred)

This is a riff on your #2.

If Aqua provided a method to bootstrap itself programmatically via Go, Atmos could invoke that as part of its process. This approach would enable similar functionality, but with a few limitations. Specifically, we’d be constrained to using Aqua’s configuration files without the ability to embed or extend those configurations directly within Atmos.

This limitation concerns me because we’re introducing radical improvements to Atmos configuration handling in an upcoming PR that adds support for remote configurations, local overrides, imports, and more. All of that provides a consistent configuration interface for atmos, including the underlying tooling. I’d like those improvements to be universally applicable across Atmos functionality.

• Refine Atmos Configuration Handling to Support Imports, Remote Imports, and Configuration Directories #808

[suzuki-shunsuke]

Sorry for late reply.
About the option 5,

1. We need to add several public API to aqua.
2. If we make atmos packages available to users directly, we need to develop something like aqua-proxy.
   We can't use aqua-proxy as aqua-proxy executes aqua exec and aqua can't use atmos configuration files.
   Please see How does Lazy Install work? too.
   We need to develop a tool like atmos-proxy and execute a command like atmos exec.
3. I considered sharing the install directory with aqua, but it may cause trouble because aqua library atmos uses might not be compatible with aqua users use.
   aqua has API to change the install directory, so atmos should change it.
4. Note that aqua uses logrus as logger.
   We can customize the setting of logrus, but probably the log format is different from the log format of atmos.
5. About checksum verification, I'm wondering how atmos supports it. This feature is important for security, so I'd like to support this somehow. But if atmos creates a file like aqua-checksums.json, this has trade-off of user experience because users need to manage the file.
6. About Renovate, we need to develop Renovate Config Preset like aqua-renovate-config. I guess this is not so difficult.

[suzuki-shunsuke]

This limitation concerns me because we’re introducing radical improvements to Atmos configuration handling in an upcoming PR that adds support for remote configurations, local overrides, imports, and more. All of that provides a consistent configuration interface for atmos, including the underlying tooling. I’d like those improvements to be universally applicable across Atmos functionality.

Probably I don't understand features remote configurations, local overrides, imports, and more completely,
but one idea is that Atmos generates aqua configuration based on those features and executes aqua.
This is like kustomize.

[osterman]

So for a general purpose package manager, aqua is exceptional. For example, I like how we can define packages that install cleanly on a local workstation as well as in CI. The proxy installer is a brilliant feature, almost like lazy loading but for binaries. Never seen that before. I like the fact that only the binaries used are the ones installed, and that they still come from the vetted sources in the aqua configuration.

What we are trying to accomplish with this, as a core capability of atmos, is so that atmos can bootstrap everything required to get off the ground. It could mean also bootstrapping aqua itself.

When we started writing Atmos, for example, we were keen to use vendir. We in fact started with it, but felt overall it was overkill for the problem we were aiming to solve. We ended up using go-getter to implement something simpler and good enough (but not for @hans-d who went for Vendir anyways!)

I feel that this maybe the case here, that if the public APIs of aqua are not yet stable for embedding, and its capabilities are so rich, it may be overkill for the scope of this feature for our purposes.

Maybe all we need is a much simpler subset of functionality? Within atmos, we have a controlled environment. Things like the Lazy Install would be easier implemented in other ways, since each command can declare its dependencies in the schema.

The goal for this feature is to make running as command in atmos automatically install the versions of the tools it needs, without any other configuration of the host system.

This includes

• helmfile commands, that can install helm and helmfile
• terraform commands that can install any OpenTofu or terraform binary
• custom commands that can install any command needed for that command to work
• workflows like custom commands can declare their dependencies which get installed

For this reason, want to stay focused on finding a way to accomplish this for a near native feel in atmos. @suzuki-shunsuke if you have any more creative ideas for this simpler use-case, I am open for it!

[suzuki-shunsuke]

If we make atmos packages available to users directly, we need to develop something like aqua-proxy.

I'd like to clarify this point.

custom commands that can install any command needed for that command to work

For instance, when Atmos installs tflint, do you expect that users can execute tflint command directly?
If so, we need to add tflint command in $PATH.
Or do you expect users can execute tflint via Atmos subcommands?
If so, we don't need to add tflint command in $PATH. This means we don't need something like aqua-proxy.

[suzuki-shunsuke]

Hmm. Please give me some time to consider this more deeply.
I want to utilize aqua echosystem while integrating it with Atmos well.

For instance, when Atmos installs tflint, do you expect that users can execute tflint command directly?

Probably you would expect this.

[osterman]

For instance, when Atmos installs tflint, do you expect that users can execute tflint command directly?

Tools like aqua, tenv, asdf, and many others follow a similar pattern. We're not trying to replace them, nor do Atmos users have to define their dependencies within Atmos if they prefer other solutions.

But the design goal of Atmos isn’t to be just another cog in an existing tool’s wheel—it’s to be the wheel itself. The goal is to establish a consistent automation environment, backed by configuration.

The Unix philosophy teaches us to "do one thing well," but the reality is that there are too many tools (too many cogs) that do one thing well—and no unified way to tie them together—the wheel.

That’s where Atmos comes in.

If your project depends on tflint (a cog), that's great. Install it with Atmos, then integrate it into your project’s configuration in a consistent, documented manner. No guesswork, no scattered tools—just a streamlined, repeatable workflow.

Developers don’t need to manually configure their shell, install 20 different tools by hand, or wrangle disparate dependencies. Instead, they install Atmos, and it all just works. Your team defines how the tools are used, and Atmos acts as the interface—making it feel as if Atmos itself implements everything.

Only, it doesn’t reinvent the wheel—it’s built on the shoulders of giants.

[osterman]

For instance, when Atmos installs tflint, do you expect that users can execute tflint command directly?
Probably you would expect this.

Actually, no. While we could optimize for that later, our real focus is on using the command within Atmos.

The goal of Atmos is that no one should need to write a shell script just to call Atmos—if they do, we've failed. No one should need to rely on Makefiles, Taskfiles, or other task runner tools—if they do, we've failed.

Atmos itself should be the interface that seamlessly integrates tools and workflows, providing a superior DX, eliminating the need for ad-hoc scripting or additional tooling just to glue things together.

[suzuki-shunsuke]

To enable users to execute installed tools in workflows, there are several options.

1. Introduce proxy like aqua-proxy for changing tool versions dynamically
2. Change environment variable $PATH for changing tool versions dynamically
3. Install tools in the directory specific to each project

About point 2, this approach is entirely different from aqua, making it difficult to reuse aqua's resources. I’d like to avoid this.

Regarding point 3, it's simple and not necessarily a bad approach. However, we would need to install tools for each project and reinstall them every time tool versions change. This results in wasted time and storage.

[suzuki-shunsuke]

How about this?

1. Develop atmos-proxy like aqua-proxy https://github.com/suzuki-shunsuke/atmos-proxy
2. Add the command atmos exec (Or atmos packages exec)
3. atmos packages install reads atmos.yaml and stack configurations and install tools into $ATMOS_PKG_DIR. The path is configurable. It creates links to atmos-proxy in $ATMOS_PKG_DIR/bin. The mechanism is same with aqua
4. atmos sub commands adds $ATMOS_PKG_DIR/bin to $PATH before executing commands
5. When commands managed by atmos are executed, atmos-proxy executes atmos exec -- <command> [<arg> ...]. atmos exec reads configuration files and installs commands before executing them

[suzuki-shunsuke]

I've created atmos-proxy based on aqua-proxy. https://github.com/suzuki-shunsuke/atmos-proxy
And I'm working on implementing aqua packages install command.
I'm considering a Go package wrapping aqua.

atmos - aquaw (wrapper) -> aqua

The wrapper abstracts aqua.
I'm not sure if it works well, but this abstraction layer may be good for both atmos and aqua.

[osterman]

Develop atmos-proxy like aqua-proxy https://github.com/suzuki-shunsuke/atmos-proxy

• Not clear how the proxy would handle multiple concurrent versions of some tool installed (e.g. opentofu, helm, helmfile, etc)

Add the command atmos exec (Or atmos packages exec)

• This makes sense to me as a feature I hadn't considered

atmos packages install reads atmos.yaml and stack configurations and install tools into $ATMOS_PKG_DIR.

• And supports multiple concurrent versions of tools

The path is configurable. It creates links to atmos-proxy in $ATMOS_PKG_DIR/bin.

• My understanding is links don't work consistently across OS-boundaries
• Links on Windows require administrative permissions
• Windows is a growing demographic of Atmos users, especially in the enterprise

When commands managed by atmos are executed, atmos-proxy executes atmos exec -- <command> [<arg> ...].

• How do you envision the version selection working?