Add native Packer support (Atmos loves Packer)

[Andriy Knysh (aknysh)]

what

• Add native HashiCorp Packer support to Atmos

• Add Packer docs and atmos packer CLI commands docs

  • https://pr-1394.atmos-docs.ue2.dev.plat.cloudposse.org/core-concepts/projects/configuration/packer/
  • https://pr-1394.atmos-docs.ue2.dev.plat.cloudposse.org/cli/commands/packer/usage/

• Add tests

why

• Use the power of Atmos components, stacks, imports, inheritance, templating and YAML functions to configure and provision Packer components to build machine images for multiple cloud platforms from Packer templates

description

Atmos natively supports HashiCorp Packer and lets you create identical machine images for multiple platforms from a single source template using the power of Atmos components, stacks, imports, inheritance, templating and YAML functions. It's compatible with every version of Packer and designed to work with multiple different versions of it concurrently.

Configure Packer in atmos.yaml

base_path: "./"

components:
  packer:
    # Can also be set using 'ATMOS_COMPONENTS_PACKER_COMMAND' ENV var, or '--packer-command' command-line argument
    command: packer
    # Can also be set using 'ATMOS_COMPONENTS_PACKER_BASE_PATH' ENV var, or '--packer-dir' command-line argument
    base_path: "components/packer"

stacks:
  base_path: "stacks"
  included_paths:
    - "deploy/**/*"
  excluded_paths:
    - "**/_defaults.yaml"
  name_template: "{{ .vars.stage }}"

Add Packer template (Packer component)

packer {
  required_plugins {
    # https://developer.hashicorp.com/packer/integrations/hashicorp/amazon
    amazon = {
      source  = "github.com/hashicorp/amazon"
      version = "~> 1"
    }
  }
}

variable "region" {
  type        = string
  description = "AWS Region"
}

variable "stage" {
  type    = string
  default = null
}

variable "ami_org_arns" {
  type        = list(string)
  description = "List of Amazon Resource Names (ARN) of AWS Organizations that have access to launch the resulting AMI(s). By default no organizations have permission to launch the AMI"
  default     = []
}

variable "ami_ou_arns" {
  type        = list(string)
  description = "List of Amazon Resource Names (ARN) of AWS Organizations organizational units (OU) that have access to launch the resulting AMI(s). By default no organizational units have permission to launch the AMI."
  default     = []
}

variable "ami_users" {
  type        = list(string)
  description = "List of account IDs that have access to launch the resulting AMI(s). By default no additional users other than the user creating the AMI has permissions to launch it."
  default     = []
}

# https://developer.hashicorp.com/packer/integrations/hashicorp/amazon#authentication
variable "assume_role_arn" {
  type        = string
  description = "Amazon Resource Name (ARN) of the IAM Role to assume. Refer to https://developer.hashicorp.com/packer/integrations/hashicorp/amazon#authentication"
}

variable "provisioner_shell_commands" {
  type        = list(string)
  description = "List of commands to execute on the machine that Packer builds"
  default     = []
}

source "amazon-ebs" "al2023" {
  ami_name      = var.ami_name
  source_ami    = var.source_ami
  instance_type = var.instance_type
  region        = var.region
  ssh_username  = var.ssh_username
  ami_org_arns  = var.ami_org_arns
  ami_ou_arns   = var.ami_ou_arns
  ami_users     = var.ami_users
  kms_key_id    = var.kms_key_arn
  encrypt_boot  = var.encrypt_boot

  force_deregister      = var.force_deregister
  force_delete_snapshot = var.force_delete_snapshot

  associate_public_ip_address = var.associate_public_ip_address

  ami_block_device_mappings {
    device_name           = "/dev/xvda"
    volume_size           = var.volume_size
    volume_type           = var.volume_type
    delete_on_termination = true
  }

  assume_role {
    role_arn         = var.assume_role_arn
    session_name     = var.assume_role_session_name
    duration_seconds = var.assume_role_duration_seconds
  }

  aws_polling {
    delay_seconds = 5
    max_attempts  = 100
  }

  tags = var.ami_tags
}

build {
  sources = ["source.amazon-ebs.al2023"]

  provisioner "shell" {
    inline = var.provisioner_shell_commands
  }

  # https://developer.hashicorp.com/packer/tutorials/docker-get-started/docker-get-started-post-processors
  # https://developer.hashicorp.com/packer/docs/post-processors
  # https://developer.hashicorp.com/packer/docs/post-processors/manifest
  post-processor "manifest" {
    output     = var.manifest_file_name
    strip_path = var.manifest_strip_path
  }
}

Configure defaults for the Packer component in the catalog

components:
  packer:
    aws/bastion:
      settings:
        packer:
          template: "main.pkr.hcl"
          source_ami: "ami-0013ceeff668b979b"
          source_ami_name: "al2023-ami-2023.7.20250527.1-kernel-6.12-arm64"
          source_ami_description: "Amazon Linux 2023 AMI 2023.7.20250527.1 arm64 HVM kernel-6.12"
          source_ami_owner_account_id: "137112412989"
          region: "us-east-2"
          org_id: "o-xxxxxxxxx"
          org_management_account_id: "xxxxxxxxxxxx"
      metadata:
        component: aws/bastion
      vars:
        # https://masterminds.github.io/sprig/date.html
        ami_name: "bastion-al2023-{{ now | unixEpoch }}"
        source_ami: "{{ .settings.packer.source_ami }}"
        region: "{{ .settings.packer.region }}"
        ami_org_arns:
          - "arn:aws:organizations::{{ .settings.packer.org_management_account_id }}:organization/{{ .settings.packer.org_id }}"
        ami_ou_arns: []
        ami_users: []
        kms_key_arn: null
        encrypt_boot: false
        ssh_username: "ec2-user"
        associate_public_ip_address: true
        volume_type: "gp3"
        skip_create_ami: false
        manifest_file_name: "manifest.json"
        manifest_strip_path: false
        assume_role_session_name: "atmos-packer"
        assume_role_duration_seconds: 1800
        force_deregister: false
        force_delete_snapshot: false
        # SSM Agent is pre-installed on AL2023 AMIs but should be enabled explicitly as done above.
        # `dnf clean all` removes cached metadata and packages to reduce AMI size.
        # `cloud-init clean` ensures the image will boot as a new instance on the next launch.
        provisioner_shell_commands:
          # Enable and start the SSM agent (already installed by default on AL2023)
          - "sudo systemctl enable --now amazon-ssm-agent"
          # Install packages, clean metadata and cloud-init
          - "sudo -E bash -c 'dnf install -y jq && dnf clean all && cloud-init clean'"
          # Install other packages
        ami_tags:
          SourceAMI: "{{ .settings.packer.source_ami }}"
          SourceAMIName: "{{ .settings.packer.source_ami_name }}"
          SourceAMIDescription: "{{ .settings.packer.source_ami_description }}"
          SourceAMIOwnerAccountId: "{{ .settings.packer.source_ami_owner_account_id }}"
          ScanStatus: pending

Define Atmos nonprod and prod stacks

vars:
  stage: nonprod

import:
  - catalog/aws/bastion/defaults

components:
  packer:
    aws/bastion:
      vars:
        # Define the variables specific to the `nonprod` account
        instance_type: "t4g.small"
        volume_size: 8
        assume_role_arn: "arn:aws:iam::NONPROD_ACCOUNT_ID:role/ROLE_NAME"
        ami_tags:
          Stage: nonprod

vars:
  stage: prod

import:
  - catalog/aws/bastion/defaults

components:
  packer:
    aws/bastion:
      vars:
        # Define the variables specific to the `prod` account
        instance_type: "t4g.medium"
        volume_size: 16
        assume_role_arn: "arn:aws:iam::PROD_ACCOUNT_ID:role/ROLE_NAME"
        ami_tags:
          Stage: prod

Execute Atmos Packer commands

> atmos packer version

Packer v1.14.1

# https://developer.hashicorp.com/packer/docs/commands/validate

> atmos packer validate aws/bastion -s nonprod

The configuration is valid.

# https://developer.hashicorp.com/packer/docs/commands/inspect

> atmos packer inspect aws/bastion -s nonprod

Packer Inspect: HCL2 mode

> input-variables:

var.ami_name: "bastion-al2023-1754457104"
var.ami_org_arns: "[\n  \"arn:aws:organizations::xxxxxxxxxxxx:organization/o-xxxxxxxxx\",\n]"
var.ami_ou_arns: "[]"
var.ami_tags: "{\n  \"ScanStatus\" = \"pending\"\n  \"SourceAMI\" = \"ami-0013ceeff668b979b\"\n  \"SourceAMIDescription\" = \"Amazon Linux 2023 AMI 2023.7.20250527.1 arm64 HVM kernel-6.12\"\n  \"SourceAMIName\" = \"al2023-ami-2023.7.20250527.1-kernel-6.12-arm64\"\n  \"SourceAMIOwnerAccountId\" = \"137112412989\"\n  \"Stage\" = \"nonprod\"\n}"
var.ami_users: "[]"

> local-variables:

> builds:

  > <0>:
    sources:
      amazon-ebs.al2023

    provisioners:
      shell

    post-processors:
      0:
        manifest

# https://developer.hashicorp.com/packer/docs/commands/init

> atmos packer init aws/bastion -s nonprod

Installed plugin github.com/hashicorp/amazon v1.3.9 in "~/.config/packer/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v1.3.9_x5.0_darwin_arm64"

# https://developer.hashicorp.com/packer/docs/commands/build

> atmos packer build aws/bastion -s nonprod

amazon-ebs.al2023:

==> amazon-ebs.al2023: Prevalidating any provided VPC information
==> amazon-ebs.al2023: Prevalidating AMI Name: bastion-al2023-1754025080
==> amazon-ebs.al2023: Found Image ID: ami-0013ceeff668b979b
==> amazon-ebs.al2023: Setting public IP address to true on instance without a subnet ID
==> amazon-ebs.al2023: No VPC ID provided, Packer will use the default VPC
==> amazon-ebs.al2023: Inferring subnet from the selected VPC "vpc-xxxxxxx"
==> amazon-ebs.al2023: Set subnet as "subnet-xxxxxxx"
==> amazon-ebs.al2023: Creating temporary keypair: packer_688c4c79-f14a-b77e-ca1e-b5b4c17b4581
==> amazon-ebs.al2023: Creating temporary security group for this instance: packer_688c4c7b-3f16-69f9-0c39-88a3fcbe94fd
==> amazon-ebs.al2023: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> amazon-ebs.al2023: Launching a source AWS instance...
==> amazon-ebs.al2023: changing public IP address config to true for instance on subnet "subnet-xxxxxxx"
==> amazon-ebs.al2023: Instance ID: i-0b621ca091aa4c240
==> amazon-ebs.al2023: Waiting for instance (i-0b621ca091aa4c240) to become ready...
==> amazon-ebs.al2023: Using SSH communicator to connect: 18.222.63.67
==> amazon-ebs.al2023: Waiting for SSH to become available...
==> amazon-ebs.al2023: Connected to SSH!
==> amazon-ebs.al2023: Provisioning with shell script: /var/folders/rt/fqmt0tmx3fs1qfzbf3qxxq700000gn/T/packer-shell653292668
==> amazon-ebs.al2023: Waiting for process with pid 2085 to finish.
==> amazon-ebs.al2023: Amazon Linux 2023 Kernel Livepatch repository   154 kB/s |  16 kB     00:00
==> amazon-ebs.al2023: Package jq-1.7.1-49.amzn2023.0.2.aarch64 is already installed.
==> amazon-ebs.al2023: Dependencies resolved.
==> amazon-ebs.al2023: Nothing to do.
==> amazon-ebs.al2023: Complete!
==> amazon-ebs.al2023: 17 files removed
==> amazon-ebs.al2023: Stopping the source instance...
==> amazon-ebs.al2023: Stopping instance
==> amazon-ebs.al2023: Waiting for the instance to stop...
==> amazon-ebs.al2023: Creating AMI bastion-al2023-1754025080 from instance i-0b621ca091aa4c240
==> amazon-ebs.al2023: Attaching run tags to AMI...
==> amazon-ebs.al2023: AMI: ami-0b2b3b68aa3c5ada8
==> amazon-ebs.al2023: Waiting for AMI to become ready...
==> amazon-ebs.al2023: Skipping Enable AMI deprecation...
==> amazon-ebs.al2023: Skipping Enable AMI deregistration protection...
==> amazon-ebs.al2023: Modifying attributes on AMI (ami-0b2b3b68aa3c5ada8)...
==> amazon-ebs.al2023: Modifying: ami org arns
==> amazon-ebs.al2023: Modifying attributes on snapshot (snap-09ad35550e1438fb2)...
==> amazon-ebs.al2023: Adding tags to AMI (ami-0b2b3b68aa3c5ada8)...
==> amazon-ebs.al2023: Tagging snapshot: snap-09ad35550e1438fb2
==> amazon-ebs.al2023: Creating AMI tags
==> amazon-ebs.al2023: Adding tag: "Stage": "nonprod"
==> amazon-ebs.al2023: Adding tag: "ScanStatus": "pending"
==> amazon-ebs.al2023: Adding tag: "SourceAMI": "ami-0013ceeff668b979b"
==> amazon-ebs.al2023: Adding tag: "SourceAMIDescription": "Amazon Linux 2023 AMI 2023.7.20250527.1 arm64 HVM kernel-6.12"
==> amazon-ebs.al2023: Adding tag: "SourceAMIName": "al2023-ami-2023.7.20250527.1-kernel-6.12-arm64"
==> amazon-ebs.al2023: Adding tag: "SourceAMIOwnerAccountId": "137112412989"
==> amazon-ebs.al2023: Creating snapshot tags
==> amazon-ebs.al2023: Terminating the source AWS instance...
==> amazon-ebs.al2023: Cleaning up any extra volumes...
==> amazon-ebs.al2023: No volumes to clean up, skipping
==> amazon-ebs.al2023: Deleting temporary security group...
==> amazon-ebs.al2023: Deleting temporary keypair...
==> amazon-ebs.al2023: Running post-processor:  (type manifest)
Build 'amazon-ebs.al2023' finished after 3 minutes 39 seconds.

==> Wait completed after 3 minutes 39 seconds

==> Builds finished. The artifacts of successful builds are:
--> amazon-ebs.al2023: AMIs were created:
us-east-2: ami-0b2b3b68aa3c5ada8

--> amazon-ebs.al2023: AMIs were created:
us-east-2: ami-0b2b3b68aa3c5ada8

# `atmos packer output` command is specific to Atmos (Packer itself does not have an `output` command)
# The command is used to get an output from a Packer manifest
# The manifest is generated by Packer when executing a `packer build` command

> atmos packer output aws/bastion -s nonprod

builds:
- artifact_id: us-east-2:ami-0c2ca16b7fcac7529
  build_time: 1.753281956e+09
  builder_type: amazon-ebs
  custom_data: null
  files: null
  name: al2023
  packer_run_uuid: 5114a723-92f6-060f-bae4-3ac2d0324557
- artifact_id: us-east-2:ami-0b2b3b68aa3c5ada8
  build_time: 1.7540253e+09
  builder_type: amazon-ebs
  custom_data: null
  files: null
  name: al2023
  packer_run_uuid: a57874d1-c478-63d7-cfde-9d91e513eb9a
  last_run_uuid: a57874d1-c478-63d7-cfde-9d91e513eb9a

# `atmos packer output` command is specific to Atmos (Packer itself does not have an `output` command)
# The command is used to get an output from a Packer manifest
# The manifest is generated by Packer when executing a `packer build` command

# Use a YQ expression to get a specific section or attribute from the Packer manifest,
# in this case, the `artifact_id` from the first build.

> atmos packer output aws/bastion -s nonprod --query '.builds[0].artifact_id'

us-east-2:ami-0c2ca16b7fcac7529

# `atmos packer output` command is specific to Atmos (Packer itself does not have an `output` command).
# The command is used to get an output from a Packer manifest.
# The manifest is generated by Packer when executing a `packer build` command.

# Use a YQ expression to get a specific section or attribute from the Packer manifest,
# in this case, the AMI (second part after the `:`) from the `artifact_id` from the first build.

> atmos packer output aws/bastion -s nonprod -q '.builds[0].artifact_id | split(":")[1]'

ami-0c2ca16b7fcac7529

[RoseSecurity]

This is awesome! Excited to test this out!

[github-actions]

These changes were released in v1.186.0.