Inconsistent final plan when using custom_origin in Cloudfront

[gnouts]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

> terraform -v
Terraform v0.12.17
+ provider.aws v2.41.0

Affected Resource(s)

• CloudFront

Terraform Configuration Files

locals {
  profile = "default"
  region = "eu-west-3"
  cidr = "10.0.0.0/16"
}

provider "aws" {
  profile    = local.profile
  region     = local.region
}

# this create a VPC with 2 subnets : one public+IGW, one private. they communicate through NAT.
module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
  name = "TEST"
  cidr = local.cidr
  azs = ["${local.region}a"]
  public_subnets  = [cidrsubnet(local.cidr,8,1)]
  private_subnets = [cidrsubnet(local.cidr,8,101)]
  enable_nat_gateway = true
  single_nat_gateway = true
  enable_vpn_gateway = false
}

data "aws_iam_instance_profile" "default" {
  name = "aws-elasticbeanstalk-ec2-role"
}

resource "aws_elastic_beanstalk_application" "backend" {
  name        = "TEST"
  description = ""
}

resource "aws_elastic_beanstalk_environment" "backend" {
  name                = "TEST"
  application         = aws_elastic_beanstalk_application.backend.name
  solution_stack_name = "64bit Amazon Linux 2018.03 v2.12.17 running Docker 18.06.1-ce"

  setting {
    resource  = ""
    namespace = "aws:ec2:vpc"
    name      = "VPCId"
    value     = module.vpc.vpc_id
  }
  setting {
    resource  = ""
    namespace = "aws:ec2:vpc"
    name      = "subnets"
    value     = join(",", module.vpc.public_subnets)
  }
  setting {
    resource  = ""
    namespace = "aws:ec2:vpc"
    name      = "elbsubnets"
    value     = join(",", module.vpc.private_subnets)
  }
  setting {
    resource  = ""
    namespace = "aws:autoscaling:launchconfiguration"
    name = "IamInstanceProfile"
    value = data.aws_iam_instance_profile.default.name
  }
}

resource "aws_cloudfront_origin_access_identity" "s3_origin_access_identity" {
  comment = "TEST"
}

resource "aws_s3_bucket" "front" {
  bucket = "terraform-test-front"
  force_destroy = true
  policy = <<EOF
{
  "Version":"2012-10-17",
  "Statement":[
    {
            "Sid": "CloudFrontRead",
            "Effect": "Allow",
            "Principal": {
                "AWS": "${aws_cloudfront_origin_access_identity.s3_origin_access_identity.iam_arn}"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::terraform-test-front/*"
        }
  ]
}
EOF
  website {
    index_document = "index.html"
    error_document = "404.html"
  }
  versioning {
      enabled = false
  }
}

data "aws_elb" "beanstalk_elb" {
  name = aws_elastic_beanstalk_environment.backend.load_balancers[0]
}

resource "aws_cloudfront_distribution" "cdn" {
  enabled      = true
  price_class  = "PriceClass_100"
  http_version = "http2"
  default_root_object = "index.html"
  default_cache_behavior {
    allowed_methods        = ["GET", "HEAD", "DELETE", "OPTIONS", "PATCH", "POST", "PUT"]
    cached_methods         = ["GET", "HEAD"]
    #min_ttl                = "0"
    #default_ttl            = "360"
    #max_ttl                = "1200"
    target_origin_id       = aws_s3_bucket.front.id
    viewer_protocol_policy = "redirect-to-https"
    compress               = true
    forwarded_values {
      query_string = false
      cookies {
        forward = "none"
      }
    }
  }
  # front
  origin {
    origin_id   = aws_s3_bucket.front.id
    domain_name = aws_s3_bucket.front.bucket_regional_domain_name
    s3_origin_config {
      origin_access_identity  = aws_cloudfront_origin_access_identity.s3_origin_access_identity.cloudfront_access_identity_path
    }
  }
  # back
  origin { 
    origin_id   = data.aws_elb.beanstalk_elb.name
    domain_name = data.aws_elb.beanstalk_elb.dns_name
    custom_origin_config {
      http_port              = 80
      https_port             = 443
      origin_protocol_policy = "https-only"
      origin_ssl_protocols   = ["TLSv1", "SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"]
    }
  }
  custom_error_response {
    error_code            = "404"
    error_caching_min_ttl = "360"
  }
  restrictions {
    geo_restriction {
      restriction_type = "none"
    }
  }
  viewer_certificate {
    cloudfront_default_certificate = true
  }
}

Debug Output

https://gist.github.com/gnouts/1a7581816e3c79a2d6779c28c733d268

Panic Output

aws_elastic_beanstalk_environment.backend: Modifications complete after 46s [id=e-kpjkj9ngk8]
data.aws_elb.beanstalk_elb: Refreshing state...

Error: Provider produced inconsistent final plan

When expanding the plan for aws_cloudfront_distribution.cdn to include new
values learned so far during apply, provider "aws" produced an invalid new
value for .origin: planned set element
cty.ObjectVal(map[string]cty.Value{"custom_header":cty.SetValEmpty(cty.Object(map[string]cty.Type{"name":cty.String,
"value":cty.String})),
"custom_origin_config":cty.ListValEmpty(cty.Object(map[string]cty.Type{"http_port":cty.Number,
"https_port":cty.Number, "origin_keepalive_timeout":cty.Number,
"origin_protocol_policy":cty.String, "origin_read_timeout":cty.Number,
"origin_ssl_protocols":cty.Set(cty.String)})),
"domain_name":cty.StringVal("terraform-test-front.s3.eu-west-3.amazonaws.com"),
"origin_id":cty.StringVal("terraform-test-front"),
"origin_path":cty.NullVal(cty.String),
"s3_origin_config":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"origin_access_identity":cty.StringVal("origin-access-identity/cloudfront/E1IP882NL2S666")})})})
does not correlate with any element in actual.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Expected Behavior

CloudFront should have been updated correctly.

Actual Behavior

Terraform crashes and does not complete other action.

Steps to Reproduce

1. Copy the tf paste above.
2. terraform init
3. terraform apply -> this one should be ok.
4. terraform apply -> Second run crashes !

Important Factoids

In my VPC, I create a public subnet that contains the Beanstalk's ELB and a private subnet that contains the Beanstalk's EC2.

I'm using CloudFront to serve my static front from S3 and redirect API calls to Beanstalk's ELB.

If you comment the second origin (custom origin redirecting to beanstalk) in cloudfront everything work fine.

References

I have reported another issue, which this one is an extension using the same code base, so might be related.
#11217

[gnouts]

I'm now pretty confident this bug is induced by #11217
If you separate beanstalk and cloudfront in two projects (two folders, two terraform apply, two tfstates) and provide variables in locals, Cloudfront doesn't crash.
I tried to used data to pass elb_subnet to cloudfront but somehow the data is marked as changing too. So you have to really use two projects to "lose" the changing state ("known after applied").
Here is the code to test it : https://gist.github.com/gnouts/40a20c986b202633da334a7246e47337

[justinretzolk]

Hey @gnouts 👋 Thank you for taking the time to file this issue and for the additional update. Given that there's been a number of AWS provider releases since your last update, can you confirm whether you're still experiencing this issue?