Merge pull request #131 from Ensono/feature-cloudwatch-addon

feat-7396: add cloudwatch addon feature for onboarding logs and metrics

aws/modules/infrastructure_modules/eks/README.md

@@ -41,7 +41,8 @@ and: https://github.com/terraform-aws-modules/terraform-aws-eks/issues/920
 
 | Name | Source | Version |
 |------|--------|---------|
-| <a name="module_eks"></a> [eks](#module\_eks) | terraform-aws-modules/eks/aws | ~> 19.20 |
+| <a name="module_container_insights_irsa_iam_role"></a> [container\_insights\_irsa\_iam\_role](#module\_container\_insights\_irsa\_iam\_role) | ../eks_irsa | n/a |
+| <a name="module_eks"></a> [eks](#module\_eks) | terraform-aws-modules/eks/aws | ~> 20.19 |
 | <a name="module_eks_kms_key"></a> [eks\_kms\_key](#module\_eks\_kms\_key) | ../../resource_modules/identity/kms_key | n/a |
 
 ## Resources
@@ -58,32 +59,40 @@ and: https://github.com/terraform-aws-modules/terraform-aws-eks/issues/920
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
 | <a name="input_cis_bootstrap_image"></a> [cis\_bootstrap\_image](#input\_cis\_bootstrap\_image) | CIS Bootstrap image, required if enable\_cis\_bootstrap is set to true | `string` | `""` | no |
+| <a name="input_cluster_addon_container_insights_config"></a> [cluster\_addon\_container\_insights\_config](#input\_cluster\_addon\_container\_insights\_config) | The configuration for the Container Insights Addon 'amazon-cloudwatch-observability'. Addon version is is tied to the Kubernetes Version. See: `aws eks describe-addon-versions --kubernetes-version <version> --addon-name 'amazon-cloudwatch-observability'` for available versions | `map(any)` | <pre>{<br>  "addon_version": "v1.8.0-eksbuild.1"<br>}</pre> | no |
+| <a name="input_cluster_addon_enable_container_insights"></a> [cluster\_addon\_enable\_container\_insights](#input\_cluster\_addon\_enable\_container\_insights) | Whether to install the the Amazon CloudWatch Observability addon to the EKS cluster for Metrics and Application Log Collection | `bool` | `true` | no |
+| <a name="input_cluster_creator_admin_permissions"></a> [cluster\_creator\_admin\_permissions](#input\_cluster\_creator\_admin\_permissions) | Adds the Terraform User that creates the cluster as an administrator | `bool` | `true` | no |
+| <a name="input_cluster_enabled_log_types"></a> [cluster\_enabled\_log\_types](#input\_cluster\_enabled\_log\_types) | A list of the log types to log, see: https://docs.aws.amazon.com/eks/latest/userguide/control-plane-logs.html | `list(string)` | <pre>[<br>  "api",<br>  "audit",<br>  "authenticator",<br>  "controllerManager",<br>  "scheduler"<br>]</pre> | no |
 | <a name="input_cluster_endpoint_private_access"></a> [cluster\_endpoint\_private\_access](#input\_cluster\_endpoint\_private\_access) | Switch to enable private access | `bool` | n/a | yes |
 | <a name="input_cluster_endpoint_public_access"></a> [cluster\_endpoint\_public\_access](#input\_cluster\_endpoint\_public\_access) | Switch to enable public access | `bool` | n/a | yes |
 | <a name="input_cluster_name"></a> [cluster\_name](#input\_cluster\_name) | Name of the cluster and resources | `string` | n/a | yes |
 | <a name="input_cluster_security_group_additional_rules"></a> [cluster\_security\_group\_additional\_rules](#input\_cluster\_security\_group\_additional\_rules) | List of additional security group rules to add to the cluster security group created. Set `source_node_security_group = true` inside rules to set the `node_security_group` as source | `any` | <pre>{<br>  "egress_nodes_ephemeral_ports_tcp": {<br>    "description": "Node all egress",<br>    "from_port": 0,<br>    "protocol": "-1",<br>    "source_node_security_group": true,<br>    "to_port": 0,<br>    "type": "egress"<br>  }<br>}</pre> | no |
 | <a name="input_cluster_single_az"></a> [cluster\_single\_az](#input\_cluster\_single\_az) | Spin up the cluster in a single AZ | `bool` | n/a | yes |
-| <a name="input_cluster_version"></a> [cluster\_version](#input\_cluster\_version) | Cluster Kubernetes Version | `string` | n/a | yes |
+| <a name="input_cluster_version"></a> [cluster\_version](#input\_cluster\_version) | Cluster Kubernetes Version | `string` | `1.3` | no |
+| <a name="input_create_kms_key"></a> [create\_kms\_key](#input\_create\_kms\_key) | [ Warn: breaking-change ] Making this value false will allow passing a custom KMS key via the provider\_key\_arn configuration | `bool` | `true` | no |
 | <a name="input_eks_desired_nodes"></a> [eks\_desired\_nodes](#input\_eks\_desired\_nodes) | The initial starting number of nodes, per AZ if 'cluster\_single\_az' is false | `string` | `2` | no |
 | <a name="input_eks_maximum_nodes"></a> [eks\_maximum\_nodes](#input\_eks\_maximum\_nodes) | The maximum number of nodes in the cluster, per AZ if 'cluster\_single\_az' is false | `string` | `3` | no |
 | <a name="input_eks_minimum_nodes"></a> [eks\_minimum\_nodes](#input\_eks\_minimum\_nodes) | The minimum number of nodes in the cluster, per AZ if 'cluster\_single\_az' is false | `string` | `1` | no |
 | <a name="input_eks_node_size"></a> [eks\_node\_size](#input\_eks\_node\_size) | Configure desired no of nodes for the cluster | `string` | `"t3.small"` | no |
 | <a name="input_eks_node_tenancy"></a> [eks\_node\_tenancy](#input\_eks\_node\_tenancy) | The tenancy of the node instance to use for EKS | `string` | `"default"` | no |
 | <a name="input_eks_node_type"></a> [eks\_node\_type](#input\_eks\_node\_type) | The type of nodes to use for EKS | `string` | `"ON_DEMAND"` | no |
 | <a name="input_enable_cis_bootstrap"></a> [enable\_cis\_bootstrap](#input\_enable\_cis\_bootstrap) | Set to true to enable the CIS Boostrap, false to disable. | `bool` | `false` | no |
+| <a name="input_image_gc_high_threshold_percent"></a> [image\_gc\_high\_threshold\_percent](#input\_image\_gc\_high\_threshold\_percent) | The percent of disk usage that initiates image garbage collection by kubelet. This value must be greater than the low threshold value | `number` | `85` | no |
+| <a name="input_image_gc_low_threshold_percent"></a> [image\_gc\_low\_threshold\_percent](#input\_image\_gc\_low\_threshold\_percent) | The kubelet deletes images until disk usage reaches this value. This value must be less than the high threshold value | `number` | `80` | no |
+| <a name="input_node_security_group_additional_rules"></a> [node\_security\_group\_additional\_rules](#input\_node\_security\_group\_additional\_rules) | List of additional security group rules to add to the node security group created. Set source\_cluster\_security\_group = true inside rules to set the cluster\_security\_group as source | `any` | <pre>{<br>  "egress_all": {<br>    "cidr_blocks": [<br>      "0.0.0.0/0"<br>    ],<br>    "description": "Node all egress",<br>    "from_port": 0,<br>    "ipv6_cidr_blocks": [<br>      "::/0"<br>    ],<br>    "protocol": "-1",<br>    "to_port": 0,<br>    "type": "egress"<br>  },<br>  "ingress_self_all": {<br>    "description": "Node to node all ports/protocols",<br>    "from_port": 0,<br>    "protocol": "-1",<br>    "self": true,<br>    "to_port": 0,<br>    "type": "ingress"<br>  }<br>}</pre> | no |
+| <a name="input_node_security_group_enable_recommended_rules"></a> [node\_security\_group\_enable\_recommended\_rules](#input\_node\_security\_group\_enable\_recommended\_rules) | Determines whether to enable recommended security group rules for the node security group created. This includes node-to-node TCP ingress on ephemeral ports and allows all egress traffic | `bool` | `true` | no |
 | <a name="input_region"></a> [region](#input\_region) | AWS region | `string` | n/a | yes |
 | <a name="input_tags"></a> [tags](#input\_tags) | Map of infrastructure tags. | `map(string)` | n/a | yes |
+| <a name="input_trusted_role_arn"></a> [trusted\_role\_arn](#input\_trusted\_role\_arn) | IAM role passed to KMS Policy | `string` | `""` | no |
 | <a name="input_vpc_id"></a> [vpc\_id](#input\_vpc\_id) | The VPC ID to use for the Cluster and resources | `string` | n/a | yes |
 | <a name="input_vpc_private_subnets"></a> [vpc\_private\_subnets](#input\_vpc\_private\_subnets) | The VPC Private Subnets to place EKS nodes into | `list(string)` | n/a | yes |
-| <a name="input_node_security_group_additional_rules"></a> [node\_security\_group\_additional\_rules](#input\_node\_security\_group\_additional\_rules) | List of additional security group rules to add to the node security group created. Set `source_cluster_security_group = true` inside rules to set the cluster_security_group as source  | `any` | <pre>{<br> ingress_self_all : { <br> description : "Node to node all ports/protocols" <br> protocol    : "-1" <br> from_port   : 0 <br> to_port     : 0 <br> type        : "ingress" <br> self        : true <br> } <br> egress_all : { <br> description      : "Node all egress" <br> protocol         : "-1" <br> from_port        : 0 <br> to_port          : 0 <br> type             : "egress" <br> cidr_blocks      : ["0.0.0.0/0"] <br> ipv6_cidr_blocks = ["::/0"] } <br>}</pre> | no |
-| <a name="input_node_security_group_enable_recommended_rules"></a> [node\_security\_group\_enable\_recommended\_rules](#input\_node\_security\_group\_enable\_recommended\_rules) | Determines whether to enable recommended security group rules for the node security group created. This includes node-to-node TCP ingress on ephemeral ports and allows all egress traffic. If we want to use our own set of rules instead of recommened rules, then make this variable false and pass your own rules via `node_security_group_additional_rules` | `bool` | `true` | no |
-
 
 ## Outputs
 
 | Name | Description |
 |------|-------------|
 | <a name="output_aws_general_eks_roles"></a> [aws\_general\_eks\_roles](#output\_aws\_general\_eks\_roles) | The EKS General Role ARNs |
+| <a name="output_cluster_arn"></a> [cluster\_arn](#output\_cluster\_arn) | The Amazon Resource Name (ARN) of the cluster |
 | <a name="output_cluster_certificate_authority_data"></a> [cluster\_certificate\_authority\_data](#output\_cluster\_certificate\_authority\_data) | base64 encoded certificate data required to communicate with your cluster |
 | <a name="output_cluster_endpoint"></a> [cluster\_endpoint](#output\_cluster\_endpoint) | Endpoint for EKS control plane. |
 | <a name="output_cluster_id"></a> [cluster\_id](#output\_cluster\_id) | EKS cluster ID. |
@@ -92,6 +101,5 @@ and: https://github.com/terraform-aws-modules/terraform-aws-eks/issues/920
 | <a name="output_cluster_oidc_provider"></a> [cluster\_oidc\_provider](#output\_cluster\_oidc\_provider) | OpenID Connect identity provider without leading http |
 | <a name="output_cluster_oidc_provider_arn"></a> [cluster\_oidc\_provider\_arn](#output\_cluster\_oidc\_provider\_arn) | OpenID Connect identity provider ARN |
 | <a name="output_cluster_security_group_id"></a> [cluster\_security\_group\_id](#output\_cluster\_security\_group\_id) | Security group ids attached to the cluster control plane. |
-| <a name="output_config_map_aws_auth"></a> [config\_map\_aws\_auth](#output\_config\_map\_aws\_auth) | A kubernetes configuration to authenticate to this EKS cluster. |
 | <a name="output_region"></a> [region](#output\_region) | AWS region |
 <!-- END_TF_DOCS -->

aws/modules/infrastructure_modules/eks/eks-cluster.tf

@@ -32,10 +32,10 @@ module "eks" {
   node_security_group_additional_rules         = var.node_security_group_additional_rules
   node_security_group_enable_recommended_rules = var.node_security_group_enable_recommended_rules
 
-  iam_role_additional_policies = var.cluster_iam_role_additional_policies
-
   cluster_enabled_log_types = var.cluster_enabled_log_types
 
+  cluster_addons = local.cluster_addons
+
   create_kms_key         = var.create_kms_key
   kms_key_administrators = var.trusted_role_arn == "" ? [] : ["${data.aws_caller_identity.this.arn}", "${var.trusted_role_arn}"]
 

aws/modules/infrastructure_modules/eks/examples/main.tf

@@ -31,10 +31,6 @@ module "eks" {
   cluster_endpoint_private_access = true
   cluster_single_az               = false
 
-  cluster_iam_role_additional_policies = {
-    cloudwatch_agent_server_policy = data.aws_iam_policy.cloudwatch_agent_server_policy.arn
-  }
-
   vpc_id              = module.vpc.id
   vpc_private_subnets = module.vpc.private_subnet_ids
 

aws/modules/infrastructure_modules/eks/iam.tf

@@ -0,0 +1,17 @@
+module "container_insights_irsa_iam_role" {
+  count = var.cluster_addon_enable_container_insights ? 1 : 0
+
+  source = "../eks_irsa"
+
+  cluster_name            = var.cluster_name
+  cluster_oidc_issuer_url = module.eks.cluster_oidc_issuer_url
+  aws_account_id          = local.account_id
+
+  policy_prefix = "/${var.cluster_name}/"
+
+  # See: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-prerequisites.html
+  namespace                        = "amazon-cloudwatch"
+  service_account_name             = "cloudwatch-agent"
+  additional_service_account_names = ["amazon-cloudwatch-observability-controller-manager"]
+  additional_policies              = ["CloudWatchAgentServerPolicy"]
+}

aws/modules/infrastructure_modules/eks/locals.tf

@@ -1,8 +1,18 @@
 locals {
-
   ## Cluster
   cluster_azs = var.cluster_single_az ? [data.aws_availability_zones.available.names[0]] : data.aws_availability_zones.available.names
 
+  cluster_container_insights_addon = var.cluster_addon_enable_container_insights ? {
+    amazon-cloudwatch-observability = merge(
+      {
+        service_account_role_arn = module.container_insights_irsa_iam_role.0.irsa_role_arn
+      },
+      var.cluster_addon_container_insights_config
+    )
+  } : {}
+
+  cluster_addons = merge(local.cluster_container_insights_addon)
+
   eks_bootstrap_extra_args = <<-EOT
   [settings.kernel]
   lockdown = "integrity"
@@ -95,4 +105,6 @@ locals {
       "Name" = local.logging_bucket_kms_key_name
     })
   )
+
+  account_id = data.aws_caller_identity.this.account_id
 }

aws/modules/infrastructure_modules/eks/variables.tf

@@ -24,9 +24,12 @@ variable "cluster_name" {
   description = "Name of the cluster and resources"
 }
 
+# NOTE: If you change this, please change the version in 'cluster_addon_container_insights_config'
 variable "cluster_version" {
   type        = string
   description = "Cluster Kubernetes Version"
+
+  default = 1.30
 }
 
 variable "cluster_endpoint_private_access" {
@@ -58,11 +61,20 @@ variable "cluster_enabled_log_types" {
   default = ["api", "audit", "authenticator", "controllerManager", "scheduler"]
 }
 
-variable "cluster_iam_role_additional_policies" {
-  type        = map(string)
-  description = "Additional policies to be added to the IAM role for the EKS Cluster"
+variable "cluster_addon_enable_container_insights" {
+  type        = bool
+  description = "Whether to install the the Amazon CloudWatch Observability addon to the EKS cluster for Metrics and Application Log Collection"
 
-  default = {}
+  default = true
+}
+
+variable "cluster_addon_container_insights_config" {
+  type        = map(any)
+  description = "The configuration for the Container Insights Addon 'amazon-cloudwatch-observability'. Addon version is is tied to the Kubernetes Version. See: `aws eks describe-addon-versions --kubernetes-version <version> --addon-name 'amazon-cloudwatch-observability'` for available versions"
+
+  default = {
+    addon_version = "v1.8.0-eksbuild.1"
+  }
 }
 
 variable "eks_minimum_nodes" {

aws/modules/infrastructure_modules/eks_irsa/README.md

@@ -39,22 +39,26 @@ No modules.
 |------|------|
 | [aws_iam_policy.policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
 | [aws_iam_role.role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role) | resource |
+| [aws_iam_role_policy_attachment.additional_attach](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
 | [aws_iam_role_policy_attachment.attach](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
+| [aws_iam_policy.additional_policies](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy) | data source |
 | [aws_iam_policy_document.role](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document) | data source |
 
 ## Inputs
 
 | Name | Description | Type | Default | Required |
 |------|-------------|------|---------|:--------:|
+| <a name="input_additional_policies"></a> [additional\_policies](#input\_additional\_policies) | A set of pre-exisiting AWS Policies to apply to the IRSA role, e.g. CloudWatchAgentServerPolicy | `set(string)` | n/a | yes |
+| <a name="input_additional_service_account_names"></a> [additional\_service\_account\_names](#input\_additional\_service\_account\_names) | Addiotional Service Accounts allowed to assume this role | `set(string)` | n/a | yes |
 | <a name="input_aws_account_id"></a> [aws\_account\_id](#input\_aws\_account\_id) | AWS account id to configure irsa role | `string` | n/a | yes |
 | <a name="input_cluster_name"></a> [cluster\_name](#input\_cluster\_name) | Name of Kubernetes cluster | `string` | n/a | yes |
 | <a name="input_cluster_oidc_issuer_url"></a> [cluster\_oidc\_issuer\_url](#input\_cluster\_oidc\_issuer\_url) | EKS cluster OIDC url | `string` | n/a | yes |
 | <a name="input_namespace"></a> [namespace](#input\_namespace) | Name of Kubernetes namespace | `string` | n/a | yes |
-| <a name="input_policy"></a> [policy](#input\_policy) | Policy json to apply to the irsa role | `string` | n/a | yes |
+| <a name="input_policy"></a> [policy](#input\_policy) | Policy json to apply to the irsa role | `string` | `""` | no |
 | <a name="input_policy_path"></a> [policy\_path](#input\_policy\_path) | The path to put the policy under, if not null the cluster\_name will be used as the path | `string` | `null` | no |
 | <a name="input_policy_prefix"></a> [policy\_prefix](#input\_policy\_prefix) | A prefix to use for the policies, which will be spliced with a dash. | `string` | `""` | no |
-| <a name="input_resource_description"></a> [resource\_description](#input\_resource\_description) | The description to assign to the policy and role | `string` | n/a | yes |
-| <a name="input_service_account_name"></a> [service\_account\_name](#input\_service\_account\_name) | Name of Kubernetes service account | `string` | n/a | yes |
+| <a name="input_resource_description"></a> [resource\_description](#input\_resource\_description) | The description to assign to the policy and role | `string` | `""` | no |
+| <a name="input_service_account_name"></a> [service\_account\_name](#input\_service\_account\_name) | Name of Kubernetes Service Account | `string` | n/a | yes |
 
 ## Outputs
 

aws/modules/infrastructure_modules/eks_irsa/data.tf

@@ -0,0 +1,5 @@
+data "aws_iam_policy" "additional_policies" {
+  for_each = var.additional_policies
+
+  name = each.value
+}