Providers crash when TMPDIR is > 92 characters

[ThisGuyCodes]

Terraform Version

v0.12.5

Terraform Configuration Files

This is the entire config required to reproduce.

provider "aws" {}

Debug Output

This is with terraform validate, though most commands hit the same error.

2019/07/24 14:59:04 [INFO] Terraform version: 0.12.5  
2019/07/24 14:59:04 [INFO] Go runtime version: go1.12.4
2019/07/24 14:59:04 [INFO] CLI args: []string{"/usr/local/bin/terraform", "validate"}
2019/07/24 14:59:04 [DEBUG] Attempting to open CLI config file: /home/thisguy/.terraformrc
2019/07/24 14:59:04 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2019/07/24 14:59:04 [INFO] CLI command args: []string{"validate"}
2019/07/24 14:59:04 [DEBUG] checking for provider in "."
2019/07/24 14:59:04 [DEBUG] checking for provider in "/usr/local/bin"
2019/07/24 14:59:04 [DEBUG] checking for provider in ".terraform/plugins/linux_amd64"
2019/07/24 14:59:04 [DEBUG] found provider "terraform-provider-aws_v2.20.0_x4"
2019/07/24 14:59:04 [DEBUG] checking for provider in "/home/thisguy/.terraform.d/plugins"
2019/07/24 14:59:04 [DEBUG] found valid plugin: "aws", "2.20.0", "/home/thisguy/code/provtes/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.20.0_x4"
2019/07/24 14:59:04 [DEBUG] checking for provisioner in "."
2019/07/24 14:59:04 [DEBUG] checking for provisioner in "/usr/local/bin"
2019/07/24 14:59:04 [DEBUG] checking for provisioner in ".terraform/plugins/linux_amd64"
2019/07/24 14:59:04 [DEBUG] checking for provisioner in "/home/thisguy/.terraform.d/plugins"
2019/07/24 14:59:04 [TRACE] terraform.NewContext: starting
2019/07/24 14:59:04 [TRACE] terraform.NewContext: resolving provider version selections
2019/07/24 14:59:04 [TRACE] terraform.NewContext: loading provider schemas
2019/07/24 14:59:04 [TRACE] LoadSchemas: retrieving schema for provider type "aws"
2019-07-24T14:59:04.734-0400 [INFO]  plugin: configuring client automatic mTLS
2019-07-24T14:59:04.765-0400 [DEBUG] plugin: starting plugin: path=/home/thisguy/code/provtes/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.20.0_x4 args=[/home/thisguy/code/provtes/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.20.0_x4]
2019-07-24T14:59:04.765-0400 [DEBUG] plugin: plugin started: path=/home/thisguy/code/provtes/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.20.0_x4 pid=29439
2019-07-24T14:59:04.765-0400 [DEBUG] plugin: waiting for RPC address: path=/home/thisguy/code/provtes/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.20.0_x4
2019-07-24T14:59:04.783-0400 [ERROR] plugin.terraform-provider-aws_v2.20.0_x4: plugin init error: error="listen unix /home/thisguy/code/provtes/thisdirnameisverylongandallitneedstobeistoolonganditwillmakethepr/plugin569537193: bind: invalid argument" timestamp=2019-07-24T14:59:04.782-0400
2019-07-24T14:59:04.785-0400 [DEBUG] plugin: plugin process exited: path=/home/thisguy/code/provtes/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.20.0_x4 pid=29439
2019-07-24T14:59:04.785-0400 [WARN]  plugin: plugin failed to exit gracefully

Error: Failed to instantiate provider "aws" to obtain schema: Unrecognized remote plugin message: 

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

Expected Behavior

Success! The configuration is valid.

Actual Behavior

Error: Failed to instantiate provider "aws" to obtain schema: Unrecognized remote plugin message: 

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

Steps to Reproduce

• Have the TMPDIR environment variable be 93 characters long, or longer.
• Run terraform init
• Run terraform (validate|plan|apply)

Additional Context

My build systems sets to TMPDIR to a build/run specific directory. That's how I hit this.

I have validated the same error occurs for the aws, google, chef, cloudflare, and grafana providers (probably more, but I figured that was enough). So I'm fairly certain this is an issue with the github.com/hashicorp/terraform/plugin library and not any individual provider.

[ThisGuyCodes]

Looks like it's a hard-coded limitation in the kernel :/
https://linux.die.net/man/7/unix

108 = 92 + pathsep + random_socket_name. Though actually the math seems to have it fail at 107?

I think short-term a good fix is just a better error, first from the plugin library then terraform itself.

Long-term are there other transports to fallback to?

[ThisGuyCodes]

Additionally: at least my naive attempts seem to indicate there is another way to do this. Using mkfifo and cat I don't seem to run into any path length limitations. Additionally there is the idea of using an abstract socket, though this may cause portability issues (and idk if that's available in the Go stdlib).

[kmoe]

Thanks for the bug report - it's an interesting one.

You're quite right that the problem here is the UNIX_PATH_MAX limit of 108 for socket path lengths, which would usually mean 107 + the null byte terminator. Go in fact tolerates the full length (see https://go-review.googlesource.com/c/go/+/66190/) but socket names cannot be longer than this.

It may be feasible in the future for Terraform to create an abstract socket here, but that may well require changes in hashicorp/go-plugin (the library we use for creating the socket), requiring broader review. Currently a TMPDIR of 93 characters or longer is too much of an edge case to necessitate this change, but if further reasons for changing the transport mechanism arise in the future, this will certainly be considered.

For now, Unix sockets in the TMPDIR are required to use Terraform. I'm going to leave this issue open as I agree that we should at least log a better error message when this fails. PRs are very welcome!

[basvandijk]

I also ran into this today. My build system (single-user Nix on MacOS) executes my terraform-validate job in a directory like:

/private/var/folders/5h/b11m6fxj2tqgbxwz5bgjy42c0000gn/T/nix-build-terraform-validate.drv-0 (91 bytes)

Nix also sets TMPDIR to this directory. Running terraform validate in that directory results in:

Error: Failed to instantiate provider "aws" to obtain schema: Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

Currently I can kind of work around this by setting:

export TMPDIR="$(mktemp -d /tmp/tf-validator.XXXXX)"

But accessing files outside the build directory (like /tmp) is not allowed when nix is run in the recommended sandbox mode which isolates the build from the rest of the filesystem thereby increasing reproducibility.

I would like to enable sandbox mode for my builds so it would be great if terraform didn't have restrictions in which directories you can run it.

[abermudez1718]

We utilize thought-machine's Please Build Tool during our build/runs for our mono repositories which contain Terraform, and consistently hit this same issue when attempting to run a terraform plan on our project.

The path has about 112 characters.

2021/03/02 14:18:02 [TRACE] terraform.NewContext: loading provider schemas
2021/03/02 14:18:02 [TRACE] LoadSchemas: retrieving schema for provider type "registry.terraform.io/hashicorp/aws"
2021-03-02T14:18:02.198Z [INFO]  plugin: configuring client automatic mTLS
2021-03-02T14:18:02.225Z [DEBUG] plugin: starting plugin: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.30.0/linux_amd64/terraform-provider-aws_v3.30.0_x5 args=[.terraform/plugins/registry.terraform.io/hashicorp/aws/3.30.0/linux_amd64/terraform-provider-aws_v3.30.0_x5]
2021-03-02T14:18:02.227Z [DEBUG] plugin: plugin started: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.30.0/linux_amd64/terraform-provider-aws_v3.30.0_x5 pid=748
2021-03-02T14:18:02.227Z [DEBUG] plugin: waiting for RPC address: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.30.0/linux_amd64/terraform-provider-aws_v3.30.0_x5
2021-03-02T14:18:02.264Z [ERROR] plugin.terraform-provider-aws_v3.30.0_x5: plugin init error: error="listen unix /home/jenkins/agent/workspace/eks-jenkins-gatekeeper/plz-out/tmp/module/aws/conftest._test/run_1/plugin700261059: bind: invalid argument" timestamp=2021-03-02T14:18:02.264Z

The path has 63 characters when we shorten the workspace path to /ejg. The error still occurs for bind: invalid argument.

2021/03/02 14:25:00 [TRACE] terraform.NewContext: starting
2021/03/02 14:25:00 [TRACE] terraform.NewContext: loading provider schemas
2021/03/02 14:25:00 [TRACE] LoadSchemas: retrieving schema for provider type "registry.terraform.io/hashicorp/aws"
2021-03-02T14:25:00.044Z [INFO]  plugin: configuring client automatic mTLS
2021-03-02T14:25:00.070Z [DEBUG] plugin: starting plugin: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.30.0/linux_amd64/terraform-provider-aws_v3.30.0_x5 args=[.terraform/plugins/registry.terraform.io/hashicorp/aws/3.30.0/linux_amd64/terraform-provider-aws_v3.30.0_x5]
2021-03-02T14:25:00.071Z [DEBUG] plugin: plugin started: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.30.0/linux_amd64/terraform-provider-aws_v3.30.0_x5 pid=438
2021-03-02T14:25:00.071Z [DEBUG] plugin: waiting for RPC address: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.30.0/linux_amd64/terraform-provider-aws_v3.30.0_x5
2021-03-02T14:25:00.111Z [ERROR] plugin.terraform-provider-aws_v3.30.0_x5: plugin init error: error="listen unix /ejg/plz-out/tmp/module/aws/conftest._test/run_1/plugin153984548: bind: invalid argument" timestamp=2021-03-02T14:25:00.111Z