Failed to get schema due to backend authentication

[garry-harthill-cko]

I believe the language server is reading the remote state. Remote state requires authentication.

I have a more elaborate method of authenticating against AWS which is a wrapper that rotates keys and sets environment variables, etc.

https://github.com/99designs/aws-vault

It would be a good feature if we would wrap the language servers execution in this program. Maybe a "terraform-ls.command" option so we can define the full terraform-ls command to be executed. eg:
/usr/local/bin/aws-vault exec myProfile -- terraform-ls

[radeksimko]

@garry-harthill-cko Generally the language server was designed to not require any access to credentials and other configuration that is usually provided out of bound for the exact reason that passing the right ENV variables or configuration down to Terraform is difficult from UX perspective.

The only terraform commands it runs are the following:

• terraform fmt -
• terraform version
• terraform providers schema -json

and each execution of either of these is also visible from the log (when you enable logging), e.g.

2020/06/01 08:22:00 exec.go:180: Starting /usr/local/bin/terraform ["terraform" "version"] in "/var/folders/fb/k_9ystyd08j934qcxmyv0wjc0000gp/T/"...
2020/06/01 08:22:00 exec.go:146: Waiting for command to finish ...
2020/06/01 08:22:00 exec.go:172: terraform run (/usr/local/bin/terraform ["terraform" "version"], in "/var/folders/fb/k_9ystyd08j934qcxmyv0wjc0000gp/T/", pid 19252) finished with exit code 0

2020/06/01 08:22:00 exec.go:180: Starting /usr/local/bin/terraform ["terraform" "providers" "schema" "-json"] in "/private/var/workspace/tf-test/github"...
...
2020/06/01 08:22:02 exec.go:172: terraform run (/usr/local/bin/terraform ["terraform" "providers" "schema" "-json"], in "/private/var/workspace/tf-test/github", pid 19278) finished with exit code 0

Obtaining the schema does require the workspace to be initialized, but terraform init (which would require credentials and access to the remote state) can run outside of the IDE.

I'm aware that the extension also has terraform init in the command palette - and I agree that we should reflect the reality you described there and somehow allow users to pass the configuration. That said you should be able to init outside of VSCode for the time being.

---

Do you mind sharing more details about the problem you're running into? Ideally a snippet of a log like the above?

[garry-harthill-cko]

Thanks for the quick reply.

I'm seeing this in the logs quite soon after starting vscode:

2020/06/01 10:09:43 schema_storage.go:112: error obtaining schemas: Unable to retrieve schemas: failed to get schemas: terraform (pid 17099) exited (code 1): exit status 1
stdout: ""
stderr: "\x1b[31m\n\x1b[1m\x1b[31mError: \x1b[0m\x1b[0m\x1b[1mNo valid credential sources found for AWS Provider.\n\tPlease see https://terraform.io/docs/providers/aws/index.html for more information on\n\tproviding credentials for the AWS Provider\x1b[0m\n\n\x1b[0m\x1b[0m\x1b[0m\n"

Then while editing references to a missing schema file are displayed:

2020/06/01 10:15:20 complete.go:53: HCL block found at HCL pos hcl.Pos{Line:32, Column:14, Byte:1181}
2020/06/01 10:15:20 complete.go:66: Configuration block "resource" parsed
2020/06/01 10:15:20 schema_storage.go:189: Reading "aws" resource schema
2020/06/01 10:15:20 schema_storage.go:141: Acquiring semaphore before reading schema
2020/06/01 10:15:20 rpc_logger.go:45: Error for "textDocument/completion" (ID 6): [-32098] finding completion items failed: no schema available
2020/06/01 10:15:20 server.go:204: Completed 1 requests [1.18848ms elapsed]

The workspace is initialised and all terraform actions work without errors on the command line.

Maybe I installed the extension incorrectly? I cloned into extensions directory and ran npm install.

[radeksimko]

Thank you for these useful additional details @garry-harthill-cko !

I will try to reproduce this 🔜 , but if it's a bug it would be more likely in the language server or Terraform itself, than the extension as it's the server that executes Terraform in this context. For that reason I'm going to transfer it over.

[radeksimko]

One more question - what Terraform version do you use for this configuration?

[radeksimko]

Also could you please try to run the following (in the mentioned workspace):

terraform providers schema -json

with and without your auth wrapper?

[garry-harthill-cko]

Also could you please try to run the following (in the mentioned workspace):

terraform providers schema -json

with and without your auth wrapper?

terraform -v
Terraform v0.12.25

Without the auth wrapper I get the error I described before:

Error: No valid credential sources found for AWS Provider.
        Please see https://terraform.io/docs/providers/aws/index.html for more information on
        providing credentials for the AWS Provider

With the auth wrapper it appears to work and returns a very large json document:

{"format_version":"0.1","provider_schemas":{"aws":{"provider":{"version":0,"block":{"attributes":{"access_key":{"type":"string","description":"The access key for API operations. You can retrieve this\nfrom the 'Security \u0026 Credentials' section of the AWS console.","optional":true},
...
...
}}}}}}}}

[garry-harthill-cko]

Can this schema document be downloaded separately and placed in a directory manually for use by the extension?

[radeksimko]

Thank you 🙏

It appears that terraform providers schema -json does in fact require state authentication, for some reason, which may be related to #56 🤔

You may be able to override the path to Terraform in the meantime via the following VSCode configuration (settings.json):

{
    "terraform.languageServer.args": [
        "serve",
        "-tf-exec=/auth/wrapper terraform"
    ]
}

We could also make this a more 1st-class settings in the VSCode UI in the future.

Can this schema document be downloaded separately and placed in a directory manually for use by the extension?

Not really and I don't think we'd want to support such workflow, because getting the UX right around updating it would be very tricky. Ideally the user shouldn't even need to know that there's such thing as machine-readable schema in most cases (in the happy path).

[radeksimko]

Actually the custom path flag isn't support yet, but will be in the upcoming LS version:
https://github.com/hashicorp/terraform-ls/blob/master/CHANGELOG.md#030-unreleased

[garry-harthill-cko]

I built the master branch of terraform-ls.

The -tf-exec= option does "work". However it expects a single executable I think.

I have managed to get it kind of working by calling another wrapper shell script which in turn exec's terraform with the correct environment. Not ideal.

[radeksimko]

You're right - it's not ideal.

Ideally obtaining schema shouldn't require state at all (and hence shouldn't require auth) IMO, but we can't really fix this in Terraform core retrospectively (in past versions), so we will need to have a decent solution before this is addressed in core.

However it expects a single executable I think.

Good catch! I'm not too excited about flags which expect string with spaces (or any kind of arrays), but it is a solution and think we could use something like go-shellwords to parse the string. The only caveat is that it needs some patches to work on Windows, e.g. mattn/go-shellwords#39

We can also prioritize the UI config option for setting the Terraform path in VSCode and try to construct sh -c "string" or cmd.exe /c "string" (depending on the OS) on the fly for the user, before setting it as a flag. Does that sound like a decent short-term solution?

---

(Unrelated to your use case)

There may be users which don't use wrappers like the one you mentioned, but set variables manually. I'm not sure there's any decent way we can support that, because passing environment variables between whatever process launches the IDE, the IDE, plugin system and terraform-ls is not trivial. And allowing user to set environment variables manually somewhere in settings doesn't make a great UX either, especially when they need to switch between workspaces.

Maybe we just discourage this is as a bad practice and make folks approach it either via wrappers, or by making terraform commands work out of the box (e.g. via S3 backend profile argument). 🤔

I mean even if you do store variables locally, then the sensitive ones should really be stored in something like envchain, so I'm more inclined to support that, as opposed to passing individual variables.

[paultyng]

This may just be an upstream issue in the AWS provider, see this issue among some others: hashicorp/terraform-provider-aws#13057

I misspoke, this is due to the shared auth package they use: https://github.com/hashicorp/aws-sdk-go-base