Apply fails in Azure Cloud Shell with "Error: Could not load plugin"

[ghost]

This issue was originally opened by @doherty100 as hashicorp/terraform#28354. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.14.10
+ provider registry.terraform.io/hashicorp/azurerm v2.55.0
+ provider registry.terraform.io/hashicorp/random v3.1.0

Terraform Configuration Files

provider "azurerm" {
  features {}
  subscription_id = var.subscription_id
  # client_id       = "REPLACE-WITH-YOUR-CLIENT-ID"
  # client_secret   = "REPLACE-WITH-YOUR-CLIENT-SECRET"    
  # tenant_id       = "REPLACE-WITH-YOUR-TENANT-ID"
}

output "subscription_id" {
  value = var.subscription_id
}

provider "random" {}

Debug Output

https://gist.github.com/doherty100/ad20d50bfad77df79018c01bd69eed35

Crash Output

Expected Behavior

Apply should succeed.

Actual Behavior

Error: Could not load plugin


Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

Failed to instantiate provider "registry.terraform.io/hashicorp/azurerm" to
obtain schema: Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

Steps to Reproduce

This only happens in Azure Cloud Shell (a docker container for CLI hooked up to Azure portal).

This only happens AFTER running multiple other terraform init / terraform apply operations in other directories.

Additional Context

I run this same combination of terraform init / terraform apply operations using terraform binary installation (not a container) and this problem doesn't occur.

References

• #26104

[doherty100]

IL do not believe this is an issue with the provider.

[favoretti]

Hi and thank you for reporting this. Judging from your debug output provider segfaulted for whatever reason. Sounds like it has more to do with the way you're invoking terraform than with the provider itself. Therefore I'm going to close this issue. Please let us know we can help you in any way. Feel free to join us on Terraform-Azure slack (link can be found in the readme of this repo), there's a large community that can help.

[doherty100]

@favoretti this is reproducible consistently, and Azure Cloud Shell is a very common environment for running Terraform. This makes Azure Cloud Shell basically unusable for Terraform. Heads up @tombuildsstuff .

[favoretti]

Re-opening for others to chip in.

[doherty100]

@favoretti happy to walk you through a repro let me know if you want me to set something up.

[rickyl2452]

Has there been any update on this issue? I am also receiving this error while going through a terraform class on cloudshell. It was working before up until today where I cannot get anything to work. I see in the debug logs the same error of exiting with a segmentation fault and this is with the latest version 2.90 although i was getting it with older versions as well. Was using both Azurerm 2.90.0 and template 2.2.0 plugin

created provider logger: level=trace 2021-12-31T19:12:01.481Z [INFO] provider: configuring client automatic mTLS 2021-12-31T19:12:01.523Z [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/2.90.0/linux_amd64/terraform-provider-azurerm_v2.90.0_x5 args=[.terraform/providers/registry.terraform.io/hashicorp/azurerm/2.90.0/linux_amd64/terraform-provider-azurerm_v2.90.0_x5] 2021-12-31T19:12:01.536Z [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/2.90.0/linux_amd64/terraform-provider-azurerm_v2.90.0_x5 pid=680 2021-12-31T19:12:01.536Z [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/2.90.0/linux_amd64/terraform-provider-azurerm_v2.90.0_x5 2021-12-31T19:12:01.536Z [TRACE] LoadSchemas: retrieving schema for provider type "registry.terraform.io/hashicorp/template" 2021-12-31T19:12:01.541Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/2.90.0/linux_amd64/terraform-provider-azurerm_v2.90.0_x5 pid=680 error="signal: segmentation fault" 2021-12-31T19:12:01.584Z [DEBUG] created provider logger: level=trace 2021-12-31T19:12:01.584Z [INFO] provider: configuring client automatic mTLS 2021-12-31T19:12:01.622Z [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4 args=[.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4] 2021-12-31T19:12:01.634Z [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4 pid=681 2021-12-31T19:12:01.634Z [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4 2021-12-31T19:12:02.289Z [INFO] provider.terraform-provider-template_v2.2.0_x4: configuring server automatic mTLS: timestamp=2021-12-31T19:12:02.289Z 2021-12-31T19:12:02.321Z [DEBUG] provider: using plugin: version=5 2021-12-31T19:12:02.321Z [DEBUG] provider.terraform-provider-template_v2.2.0_x4: plugin address: address=/tmp/plugin914416072 network=unix timestamp=2021-12-31T19:12:02.321Z 2021-12-31T19:12:02.387Z [TRACE] GRPCProvider: GetProviderSchema 2021-12-31T19:12:02.387Z [TRACE] provider.stdio: waiting for stdio data 2021-12-31T19:12:02.387Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio" 2021-12-31T19:12:02.392Z [DEBUG] No provider meta schema returned 2021-12-31T19:12:02.392Z [TRACE] GRPCProvider: Close 2021-12-31T19:12:02.398Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/template/2.2.0/linux_amd64/terraform-provider-template_v2.2.0_x4 pid=681 2021-12-31T19:12:02.398Z [DEBUG] provider: plugin exited

[doherty100]

@rickyl2452 I gave up on using Azure Cloud Shell for TF. Besides this issue the timeout error for long plan applications make it a "demo only" kind of environment. In researching this, there was some other folks who experienced issues with cached providers in containers, but I gave up and moved on. One thing I did change was to stop checking in terraform.lock.hcl files into my repo and start pinning my configurations to a specific version of the azurerm provider using required_providers as per Hashicorp best practices. I haven't retested on Azure Cloud Shell since but I'm curious if this would resolve the provider caching issue I was having.

[tombuildsstuff]

Unfortunately this appears to be a CloudShell issue, so I'm not sure there's anything we can do to work around it in the Provider. Whilst it may be possible to workaround this in Terraform Core, I have a feeling that what's happening is the Provider is either being significantly delayed, or hitting the memory ceiling within CloudShell - but unfortunately I can't confirm that.

Since this appears to be a CloudShell issue rather than a Terraform specific issue, I'm going to close this issue for the moment - but I'd suggest opening an issue on the CloudShell repository where someone from that team should be able to take a look.

Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.