Cloudwatch Log Group created with options does not get auto destroyed

[dwaiba]

I have issues

I'm submitting a...

• bug report
• feature request
• support request - read the FAQ first!
• kudos, thank you, warm fuzzy

What is the current behavior?

On Creating with the following options
cluster gets created nicely but on destroy it throws
**Error: Creating CloudWatch Log Group failed: ResourceAlreadyExistsException: The specified log group already exists: The CloudWatch Log Group '/aws/eks/test-eks/cluster' already exists.

on .terraform/modules/eks-cluster/terraform-aws-eks-12.1.0/cluster.tf line 1, in resource "aws_cloudwatch_log_group" "this":
1: resource "aws_cloudwatch_log_group" "this" {**

module "eks-cluster" {
  source                    = "terraform-aws-modules/eks/aws"
  create_eks                = lookup(var.eks_params, "createeks")
  cluster_name              = lookup(var.eks_params, "cluster_name")
  cluster_version           = lookup(var.eks_params, "cluster_version")
  subnets                   = module.vpc.public_subnets
  vpc_id                    = module.vpc.vpc_id
  cluster_enabled_log_types = ["api", "audit", "authenticator", "controllerManager", "scheduler"]

  enable_irsa = lookup(var.eks_params, "enable_irsa")

  worker_groups = [
    {
      name                 = "worker-group-1"
      instance_type        = "t2.medium"
      asg_desired_capacity = 1
      asg_max_size         = 5
      tags = [
        {
          "key"                 = "k8s.io/cluster-autoscaler/enabled"
          "propagate_at_launch" = "true"
          "value"               = "true"
        },
        {
          "key"                 = "k8s.io/cluster-autoscaler/${lookup(var.eks_params, "cluster_name")}"
          "propagate_at_launch" = "true"
          "value"               = "true"
        }
      ]
    },
    {
      name                 = "worker-group-2"
      instance_type        = "t2.medium"
      asg_desired_capacity = 2
      asg_max_size         = 5
      tags = [
        {
          "key"                 = "k8s.io/cluster-autoscaler/enabled"
          "propagate_at_launch" = "true"
          "value"               = "true"
        },
        {
          "key"                 = "k8s.io/cluster-autoscaler/${lookup(var.eks_params, "cluster_name")}"
          "propagate_at_launch" = "true"
          "value"               = "true"
        }
      ]
    }
  ]
}

If this is a bug, how to reproduce? Please include a code sample if relevant.

Create with the above options and destroy

What's the expected behavior?

The Destroy should take care of the Cloudwatch Log Group created with the options.

Are you able to fix this problem and submit a PR? Link here if you have already.

NA

Environment details

• Affected module version: v12.1.0
• OS:
• Terraform version:
  Terraform v0.12.24
  provider.aws v2.66.0
  provider.kubernetes v1.11.3
  provider.local v1.4.0
  provider.null v2.1.2
  provider.random v2.2.1
  provider.template v2.1.2

Any other relevant info

[milesarmstrong]

We're seeing this issue too. It seems that terraform does correctly destroy the log group, but that logs are still shipped to CloudWatch for some time after AWS say an EKS cluster has been deleted, causing CloudWatch to recreate the log group.

[milesarmstrong]

We're doing terraform destroy followed by terraform apply in two CI stages, and we've added a stage in-between the two that blocks until the log group is really gone.

We've seen it take ~3 minutes to stop being recreated.

function get_log_group {
  aws logs describe-log-groups --log-group-name-prefix ${LOG_GROUP_NAME} --query 'length(logGroups)'
}

REALLY_GONE=0

while [[ $REALLY_GONE -eq 0 ]]; do
  while [[ $(get_log_group) -gt 0 ]]; do
    echo "${LOG_GROUP_NAME} still exists, deleting it..."
    aws logs delete-log-group --log-group-name ${LOG_GROUP_NAME}
    sleep 5
  done
  echo "${LOG_GROUP_NAME} has been deleted. Waiting 300 seconds to validate it is not recreated..."
  sleep 300
  if [[ $(get_log_group) -eq 0 ]]; then
    REALLY_GONE=1
  fi
done

[maiconbaumx]

This is happening because EKS Cluster gets destroyed after Terraform delete the Cloudwatch Log Group. The AmazonEKSServicePolicy IAM policy (that is assigned to EKS Cluster role by default within this module) has permissions to CreateLogGroup and anything else needed to continue to logging correctly. When the Terraform destroys the Cloudwatch Log Group, the EKS Cluster that is running create it again. Then, when you run Terraform Apply again, the Cloudwatch Log Group doesn't exist in your state anymore (because the Terraform actually destroyed it) and the Terraform doesn't know this resource created outside him.

[stevehipwell]

Could the Terraform config be changed to make sure that the EKS cluster has been destroyed before the CloudWatch log group is destroyed?

[barryib]

This is happening because EKS Cluster gets destroyed after Terraform delete the Cloudwatch Log Group. The AmazonEKSServicePolicy IAM policy (that is assigned to EKS Cluster role by default within this module) has permissions to CreateLogGroup and anything else needed to continue to logging correctly. When the Terraform destroys the Cloudwatch Log Group, the EKS Cluster that is running create it again. Then, when you run Terraform Apply again, the Cloudwatch Log Group doesn't exist in your state anymore (because the Terraform actually destroyed it) and the Terraform doesn't know this resource created outside him.

Humm are you sure about that @maiconbaumx ? There is an explicit dependency between the cluster resource and the cloudwatch log group. Technically, Terraform should destroy aws_cloudwatch_log_group.this before aws_eks_cluster.this.

[dwaiba]

@barryib yep... that seems to be the case as in terraform-aws-modules/terraform-aws-vpc#435 and hashicorp/terraform#14750 too

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stevehipwell]

Is there any news on this?

[RavindraSinghKhichi]

I am facing the same issue. Any updates on this, expecting auto deletion of the log group once the VPC is destroyed.

[ulfox]

Fixed this by adding a depends_on to the cloudwatch log resource

[barryib]

@ulfox can please you share your code snippet.

[ulfox]

@ulfox can please you share your code snippet.

Sure

// ------------------------------------------------------------
// EKS Cluster Cloudwatch Group
// ------------------------------------------------------------
resource "aws_cloudwatch_log_group" "eks_cluster" {
      count             = local.cluster_log_types_enabled ? 1 : 0
      name              = local.eks_aws_cloudwatch_log_group
      retention_in_days = local.eks_log_retention_in_days

      tags = local.eks_cloudwatch_tags
}

// ------------------------------------------------------------
// EKS Cluster
// ------------------------------------------------------------
resource "aws_eks_cluster" "main" {
      name                      = var.eks_cluster_name
      role_arn                  = aws_iam_role.eks_cluster_role.arn
      enabled_cluster_log_types = local.cluster_log_types

      version = local.eks_version

      vpc_config {
            subnet_ids = local.eks_subnet_ids
            security_group_ids = [
                  aws_security_group.ControlPlaneSecurityGroup.id,
            ]
            endpoint_private_access = var.eks_endpoint_private_access
            endpoint_public_access  = var.eks_endpoint_public_access
      }

      timeouts {
            delete = var.eks_delete_timeouot
      }

      depends_on = [
            aws_iam_role_policy_attachment.AmazonEKSClusterPolicy,
            aws_iam_role_policy_attachment.AmazonEKSServicePolicy,
            aws_cloudwatch_log_group.eks_cluster.0
      ]
}