Create a Terraform mirror protocol datasource

[braunsonm]

How are you running Renovate?

Self-hosted Renovate

If you're self-hosting Renovate, tell us what version of Renovate you run.

35.32.2

If you're self-hosting Renovate, select which platform you are using.

Bitbucket Server

Was this something which used to work for you, and then stopped?

I am trying to get this working for the first time

Describe the bug

The logic for the Terraform Module and Provider datasource is flawed in that it uses the service discovery API that is only available on the official Terraform registries. This API is not to be used with custom registries. The documentation states this on Hashicorp here

The Provider Network Mirror protocol does not use the service discovery indirection, because a network mirror location is only a physical location and is never used as part of the identifier of a dependency in a Terraform configuration.

There are two problems with the logic in the datasource:

• First, the equality check to see if the registry is hashicorp and therefore has the extended API, does not account for configuration that might change the default registry URLs
• The service discovery API is still called even if the result is not required

See the logic here: https://github.com/renovatebot/renovate/blob/main/lib/modules/datasource/terraform-provider/index.ts#L65-L81

Given this configuration of renovate:

{
  "packageRules": [
    "matchDatasources": [
       "terraform-provider"
    ],
    "defaultRegistryUrls": [
      "https://mycustommirror.com"
    ]
  ]
}

First, the equality check in the lines I mentioned, just check the position of the registry URL in the list, instead of actually checking the contents are hashicorp. Second, service discovery is still called even if it is not required because queryRegistryExtendedApi should not be called.

The specific problem is here:

    const serviceDiscovery = await this.getTerraformServiceDiscoveryResult(
      registryUrl
    );

which is being called regardless of the registryUrl. This should only be called if the registry URL is from hashicorp, not from mirrors.

Custom solutions like Artifactory that implement the Terraform specification, rightly do not provide a .well-known/terraform.json, so any hosted registry will not work.

Relevant debug logs

Logs

{"name":"renovate","hostname":"renovate-hn7-mzgtc","msg":"GET https://myregistry.com/.well-known/terraform.json = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=16)"}

Have you created a minimal reproduction repository?

I have linked to a minimal reproduction in the description above

[secustor]

Both the provider and module registry protocols require a service discovery endpoint. If Artifactory do not exposes these, they are not following the protocol.

The network mirror protocol is simply currently not supported as datasource.

Regarding the defaultRegistryURL positions are referring to the readonly definition, so the order does not have an effect in your case.

Further I can see no linked reproduction repo.

[github-actions]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[braunsonm]

The network mirror protocol is simply currently not supported as datasource.

Any plans to change this? This means that these modules cannot be used in firewalled corporate environments.

Further I can see no linked reproduction repo

I don't really think a reproduction repo is required. I provided the configuration to reproduce the problem, I can't really provide you with a mirror that has the problem.

[secustor]

Currently there is no plan to implement this. The mirror protocol also only works for providers and not modules.

This means that these modules cannot be used in firewalled corporate environments.

This is only the case if you use Artifactory and at the same time use custom providers which are published there.

I don't really think a reproduction repo is required. I provided the configuration to reproduce the problem, I can't really provide you with a mirror that has the problem.

I need to know what the Terraform code you try to implement looks like.

[braunsonm]

Sample here: https://github.com/braunsonm/renovate-terraform-providers-bug
Added with a intentionally out of date version of the provider. Like I said you will need to change the renovate.json though with your mirror.

[secustor]

Your Terraform code does not use a network mirror.

[braunsonm]

Terraform code should not contain any registry mirror configuration. That is done on your local machine via a .terraformrc file.

The repo gives you a minimal reproduction sample where renovate is configured to use a mirror. If that mirror implements the provider mirror protocol, renovate will not work.