Terraform Cloud: Multiple 'schema: unknown provider "registry.terraform.io/-/..."' errors with official providers when upgrading to 0.13

[jnewland]

Terraform Version

Terraform v0.13.1
...
+ provider registry.terraform.io/-/random v2.3.0
...
+ provider registry.terraform.io/hashicorp/random v2.3.0

Terraform Configuration Files

terraform {
  backend "remote" {
  ...
  }
  required_version = "~> 0.13.1"
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 2.1"
    }
  ...
  }

Debug Output

Unavailable, using terraform cloud

Expected Behavior

The first apply on a new version should not fail after following the upgrade guide and obtaining a clean plan on the new version.

Actual Behavior / Reproduction steps

• Read upgrade guide
• Take all actions indicated:
  • Run terraform 0.13upgrade
  • Update provider syntax
• Run terraform init locally (returns cleanly)
• Obtain clean plan using terraform cloud
• Merge PR
• Attempt to apply using terraform cloud
• Encounter this error:

Terraform v0.13.1
Initializing plugins and modules...

Error: Could not load plugin


Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

5 problems:

- Failed to instantiate provider "registry.terraform.io/-/aws" to obtain
schema: unknown provider "registry.terraform.io/-/aws"
- Failed to instantiate provider "registry.terraform.io/-/local" to obtain
schema: unknown provider "registry.terraform.io/-/local"
- Failed to instantiate provider "registry.terraform.io/-/mongodbatlas" to
obtain schema: unknown provider "registry.terraform.io/-/mongodbatlas"
- Failed to instantiate provider "registry.terraform.io/-/random" to obtain
schema: unknown provider "registry.terraform.io/-/random"
- Failed to instantiate provider "registry.terraform.io/-/template" to obtain
schema: unknown provider "registry.terraform.io/-/template"

Additional Context

#25705 indicates that this is an expected error, which is confusing to me after reading the upgrade guide both before and after encountering it:

• The error message asks me to run terraform init, which is impossible in my workflow
• Remediation steps for similar errors are included in an in-house providers section: we are not using any in-house providers
• The FAQ entry for this point indicates that these messages should not cause workflow problems:

While this does not cause any problems for Terraform, it has been confusing.

Is it expected that users that do not use any in-house providers encounter this error? Are the remediation steps the same? I'd like to confirm before taking any destructive actions like terraform state replace-provider. Thanks in advance!

References

/cc #25705

[caronicolas]

I have the same one yesterday, I use the command "terraform state replace-provider" and then it works.

[gdurandvadas]

I'm having this exact problem but I just tried with one of my workspaces and passed. Could be they rolled-out a fix?

[jnewland]

I have the same one yesterday, I use the command "terraform state replace-provider" and then it works.

I expect that this will be the case, but I'd like to double check with someone from Hashicorp before doing so since the upgrade guide seems to specifically say that I shouldn't be experiencing any workflow problems.

I'm having this exact problem but I just tried with one of my workspaces and passed. Could be they rolled-out a fix?

I just retried my run and it failed in the exact same fashion. 😢

[gdurandvadas]

Ok, I figured why it worked. I ran those workspaces locally first and then on Terraform Cloud.
The other option is what @caronicolas mentioned, here is an example for the aws provider:

➜ terraform state replace-provider 'registry.terraform.io/-/aws' 'registry.terraform.io/hashicorp/aws'
Acquiring state lock. This may take a few moments...
Terraform will perform the following actions:

  ~ Updating provider:
    - registry.terraform.io/-/aws
    + registry.terraform.io/hashicorp/aws

Changing X resources:

  {edited}

Do you want to make these changes?
Only 'yes' will be accepted to continue.

Enter a value: yes

Successfully replaced provider for X resources.

If you see the error message, it says that can't find the provider in "registry.terraform.io/-", so replacing that with the correct HashiCorp URL, fixes the issue.

[alisdair]

Thanks for reporting this issue! I don't think you should be seeing this error.

Unfortunately, I'm unable to reproduce the problem given your steps. Here's what I did, all commands executed locally using the remote backend:

1. Created a new Terraform configuration:

   terraform {
     backend "remote" {
       workspaces {
         name = "26075"
         organization = "<redacted>"
       }
     }
   }
   
   resource "random_pet" "friend" {
   }

2. Run terraform-0.12.28 init: successfully creates Terraform Cloud workspace

3. Run terraform-0.12.28 apply: applies, creates resource, saves state

4. Run terraform-0.13.1 0.13upgrade: creates versions.tf file as expected

5. Update my Terraform Cloud workspace to use version 0.13.1 (using the UI)

6. Run terraform-0.13.1 plan: speculative plan succeeds on Terraform Cloud

7. Run terraform-0.13.1 apply: applies on Terraform Cloud, no changes

8. Changed the configuration (set length = 3 on the resource), reran apply: still works

Using these steps as a starting point, are you able to find a way to reproduce the problem? Without a simple reproduction test case, it's very difficult to figure out what could be going wrong here.

[jnewland]

Hey @alisdair! I'm unable to reproduce using those steps. I also tried the same set of steps with a versions.tf containing the following in the 0.12.28 phase:

provider "random" {
   version = "~> 2.1"
 }

That didn't reproduce the issue either.

I'm inclined to move forward with the terraform replace-provider remediation steps outlined above to unblock my environment given others experience here. But before I do, I'm going to leave behind some debugging notes that may help you or others figure out what's going on here.

terraform providers outputs the following information at the end, which seems to be a sign of the problem I'm experiencing.

Providers required by state:

    provider[registry.terraform.io/-/local]

    provider[registry.terraform.io/-/mongodbatlas]

    provider[registry.terraform.io/-/null]

    provider[registry.terraform.io/-/random]

    provider[registry.terraform.io/-/template]

    provider[registry.terraform.io/-/aws]

    provider[registry.terraform.io/-/helm]

    provider[registry.terraform.io/-/kubernetes]

terraform init also warns me that I don't have version constraints set for the un-namepaced name of the same set of modules when run locally, even though I have a full required_providers section with version constraints for each:

To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.

* -/aws: version = "~> 3.4.0"
* -/helm: version = "~> 1.2.4"
* -/kubernetes: version = "~> 1.12.0"
* -/local: version = "~> 1.4.0"
* -/mongodbatlas: version = "~> 0.6.3"
* -/null: version = "~> 2.1.2"
* -/random: version = "~> 2.3.0"
* -/template: version = "~> 2.1.2"

In addition to those explicit dependencies, my configuration also includes resources created by several modules that use the following backward-compatible form to declare provider dependencies:

https://github.com/terraform-aws-modules/terraform-aws-s3-bucket/blob/master/versions.tf

terraform {
  required_version = ">= 0.12.6, < 0.14"

  required_providers {
    aws = ">= 3.0, < 4.0"
  }
}

[jnewland]

Ahh, please ignore. This error was due to a mismatch between my local CLI environment and terraform cloud. Closing since the original issue has been resolved.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.