Support creating multiple worker auto scaling groups

[kppullin]

PR o'clock

Description

The objective of this change is to allow for defining multiple worker groups, similar to those supported by kops (https://github.com/kubernetes/kops/blob/master/docs/instance_groups.md).

A use case of multiple worker groups is to segment worker nodes by their primary function. For example, one set may be general purpose group of high-cpu instances for web apis, while another is a set of high-memory instances dedicated to data analytics.

I'll preface this with a disclaimer: I'm not well versed in terraform and quite open to suggestions on cleaning this up for an eventual merge.

To do so I've moved the various worker variables to a new worker_groups list of maps (yuck?).

variable "worker_groups" {
  description = "A list of maps defining worker group configurations."
  type        = "list"

  default = [
    {
      name                 = "nodes"    # Name of the worker group.
      ami_id               = ""         # AMI ID for the eks workers. If none is provided, Terraform will search for the latest version of their EKS optimized worker AMI.
      asg_desired_capacity = "1"        # Desired worker capacity in the autoscaling group.
      asg_max_size         = "3"        # Maximum worker capacity in the autoscaling group.
      asg_min_size         = "1"        # Minimum worker capacity in the autoscaling group.
      instance_type        = "m4.large" # Size of the workers instances.
    },
  ]
}

Further, various resources, etc. are moved to a new worker_groups module.

Additional concerns

• kops instance groups create separate security groups per instance/worker group. This is beyond my needs at the moment.
• Default value handling can be improved and/or better documented. Now that the worker vars are in a map, only a complete default configuration is applied. In other words, if asg_max_size is not set in the workger_group variable, a default value is NOT applied. Moving the default value to the lookup() function feels wrong, as that value may be referenced elsewhere in the future.
• This is a breaking change.

Checklist

• terraform fmt and terraform validate both work from the root and examples/eks_test_fixture directories (look in CI for an example)
• Tests for the changes have been added and passing (for bug fixes/features)
• Test results are pasted in this PR (in lieu of CI)
• Docs have been added/updated (for bug fixes/features)
• [ x] Any breaking changes are noted in the description above

[brandonjbjelland]

Hey @kppullin ! Thanks for taking interest in the project; this is a sizable contribution! 🙇

I like this direction and it closely resembles how I've approached the problem of creating N of related resources in my ALB module. I think there's room to do something very similar here.

I cringe a little bit at the prospect of submoduling. In fact, this afternoon I was working to pull the ebs_optimized submodule out into its own utility module as that allows for versioning and better testing isolation. Generally module nesting imposes constraints that I'd rather not introduce (lack of count, chained dependencies, no versioning) so it's worth reevaluating.

The test fixture needs to be updated to reflect what you've got here. I'll review what you've got and try to get that in line with what you have here.

[kppullin]

I appreciate the feedback @brandoconnor and I'll await your further research.

I'm not currently dependent upon multiple worker groups and will likely revert back to the master branch for the time being, but I anticipate others will also have this need as EKS & this module gain traction.

There's one or two other small enhancements that'd be handy, for which I'll open issues as I come across them next week.

[brandonjbjelland]

Hey @kppullin -

I don't think it's possible for me to commit to the branch you have running here but I went ahead and rounded out the work in a separate branch, keeping your initial commit which took on some of the initial heavy lifting. Have a look at what I've got there and let me know if that fits what you had in mind: #13

[brandonjbjelland]

I'm ready to ship that alternate branch to master and release as v1.0.0. Thanks again on the work here and fleshing out the idea/use case. You've been granted a 🥨 in the changelog.

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.