Feature Request: Add support for opentofu

[mcwarman]

Is your feature request related to a problem?

OpenTofu went GA recently.

With terraform support added in: #1554, #1576, and #1621.

Similar support for opentofu could be added.

It would probably be a case of working our the similarities and refactoring a few parts out to a shared utils around HCL processing vs Terraform specific parts.

Also identifying where alternatives should be used, for example OpenTofu have a fork https://github.com/opentofu/registry-address which is their alternative to https://github.com/hashicorp/terraform-registry-address

Solution you'd like

No response

Alternatives you've considered

No response

Anything else?

No response

[quotidian-ennui]

Currently updatecli does not work with alternative registries with respect to terraform lock files (the upstream tfupdate library is the culprit it appears).

Steps to reproduce

• Clone https://github.com/quotidian-ennui/updatecli-testbed

bsh ❯ cd terraform
bsh ❯ rm -rf .terraform .terraform.lock.hcl
bsh ❯ tofu init
...
OpenTofu has been successfully initialized!
bsh ❯ cat .terraform.lock.hcl (the original .terraform.lock.hcl has registry.terraform.io/)
...
provider "registry.opentofu.org/hashicorp/kubernetes" {
...
bsh ❯ export GITHUB_TOKEN=$(gh auth token)
bsh ❯ updatecli apply -c updatecli.d/tf-k8s.yaml
...
tf-provider-k8s
---------------
⚠ - "kubernetes" updated from "2.21.0" to "2.25.2" in file "terraform/versions.tf"

tf-lock-k8s
-----------
2024/01/11 17:28:16 [DEBUG] providerIndex.createProviderVersion: hashicorp/kubernetes, 2.25.2, linux_amd64
2024/01/11 17:28:16 [DEBUG] Client.ProviderPackageMetadata: GET https://registry.terraform.io/v1/providers/hashicorp/kubernetes/2.25.2/download/linux/amd64
2024/01/11 17:28:16 [DEBUG] ProviderDownloaderClient.download: GET https://releases.hashicorp.com/terraform-provider-kubernetes/2.25.2/terraform-provider-kubernetes_2.25.2_linux_amd64.zip
2024/01/11 17:28:16 [DEBUG] ProviderDownloaderClient.download: GET https://releases.hashicorp.com/terraform-provider-kubernetes/2.25.2/terraform-provider-kubernetes_2.25.2_SHA256SUMS
2024/01/11 17:28:17 [DEBUG] providerIndex.createProviderVersion: hashicorp/kubernetes, 2.25.2, windows_amd64
2024/01/11 17:28:17 [DEBUG] Client.ProviderPackageMetadata: GET https://registry.terraform.io/v1/providers/hashicorp/kubernetes/2.25.2/download/windows/amd64
2024/01/11 17:28:17 [DEBUG] ProviderDownloaderClient.download: GET https://releases.hashicorp.com/terraform-provider-kubernetes/2.25.2/terraform-provider-kubernetes_2.25.2_windows_amd64.zip
2024/01/11 17:28:17 [DEBUG] ProviderDownloaderClient.download: GET https://releases.hashicorp.com/terraform-provider-kubernetes/2.25.2/terraform-provider-kubernetes_2.25.2_SHA256SUMS
2024/01/11 17:28:17 [DEBUG] providerIndex.createProviderVersion: hashicorp/kubernetes, 2.25.2, darwin_amd64
2024/01/11 17:28:17 [DEBUG] Client.ProviderPackageMetadata: GET https://registry.terraform.io/v1/providers/hashicorp/kubernetes/2.25.2/download/darwin/amd64
2024/01/11 17:28:17 [DEBUG] ProviderDownloaderClient.download: GET https://releases.hashicorp.com/terraform-provider-kubernetes/2.25.2/terraform-provider-kubernetes_2.25.2_darwin_amd64.zip
2024/01/11 17:28:18 [DEBUG] ProviderDownloaderClient.download: GET https://releases.hashicorp.com/terraform-provider-kubernetes/2.25.2/terraform-provider-kubernetes_2.25.2_SHA256SUMS
2024/01/11 17:28:18 [DEBUG] providerIndex.createProviderVersion: hashicorp/kubernetes, 2.25.2, darwin_arm64
2024/01/11 17:28:18 [DEBUG] Client.ProviderPackageMetadata: GET https://registry.terraform.io/v1/providers/hashicorp/kubernetes/2.25.2/download/darwin/arm64
2024/01/11 17:28:18 [DEBUG] ProviderDownloaderClient.download: GET https://releases.hashicorp.com/terraform-provider-kubernetes/2.25.2/terraform-provider-kubernetes_2.25.2_darwin_arm64.zip
2024/01/11 17:28:19 [DEBUG] ProviderDownloaderClient.download: GET https://releases.hashicorp.com/terraform-provider-kubernetes/2.25.2/terraform-provider-kubernetes_2.25.2_SHA256SUMS

ERROR: something went wrong in target "tf-lock-k8s" : "✗ cannot find value for \"registry.terraform.io/hashicorp/kubernetes\" from file \"terraform/.terraform.lock.hcl\""

Pipeline "terraform-provider-k8s" failed
Skipping due to:
        targets stage:  "something went wrong during target execution"

The underlying library 'tfupdate' doesn't seem to support other registries since doing this

bsh ❯ tfupdate lock --platform=windows_amd64 --platform=linux_amd64 --platform=darwin_amd64 --platform=darwin_arm64 .

Actually adds the equivalent registry.terraform.io/ entries in to the lock file, so there are effectively double the number of providers in the lock file.

[mcwarman]

It looks like tfupdate might actually work.

We can provide a config that the CLI doesn't expose, doing some hacking locally to see whats possible:

updatecli/pkg/plugins/resources/terraform/lock/main.go

Lines 71 to 76 in bf6ad3e

	lockIndex, err := lock.NewDefaultIndex()
	if err != nil {
	return nil, err
	}
	newResource.lockIndex = lockIndex

-       lockIndex, err := lock.NewDefaultIndex()
+       client, err := lock.NewProviderDownloaderClient(lock.TFRegistryConfig{
+               BaseURL: "https://registry.opentofu.org/",
+       })
        if err != nil {
                return nil, err
        }
 
-       newResource.lockIndex = lockIndex
+       newResource.lockIndex = lock.NewIndex(client)

Produces

2024/01/12 11:06:18 [DEBUG] providerIndex.createProviderVersion: hashicorp/kubernetes, 2.23.0, linux_amd64
2024/01/12 11:06:18 [DEBUG] Client.ProviderPackageMetadata: GET https://registry.opentofu.org/v1/providers/hashicorp/kubernetes/2.23.0/download/linux/amd64
2024/01/12 11:06:18 [DEBUG] ProviderDownloaderClient.download: GET https://github.com/opentofu/terraform-provider-kubernetes/releases/download/v2.23.0/terraform-provider-kubernetes_2.23.0_linux_amd64.zip
2024/01/12 11:06:21 [DEBUG] ProviderDownloaderClient.download: GET https://github.com/opentofu/terraform-provider-kubernetes/releases/download/v2.23.0/terraform-provider-kubernetes_2.23.0_SHA256SUMS

[quotidian-ennui]

Does this suggest a new optional config a-la

 spec:
      file: terraform/.terraform.lock.hcl
      provider: hashicorp/kubernetes
      registry_url: https://registry.opentofu.org (default to registry.terraform.io)

[mcwarman]

@quotidian-ennui you might be interested in testing #1870

Went with slightly different config, which played very nicely with how it had already been coded:

targets:
  provider:
    name: 'Bump hashicorp/hcp to {{ source "latestRelease" }}'
    kind: terraform/lock
    sourceid: latestRelease
    spec:
      file: .terraform.lock.hcl
      provider: registry.opentofu.org/hashicorp/hcp
      skipconstraints: true
      platforms:
        - darwin_amd64
        - linux_amd64
        - windows_amd64

[mcwarman]

terraform/provider support still needs some work, The endpoint that is currently used isn't available on registry.opentofu.org

https://registry.terraform.io/v1/providers/hashicorp/kubernetes

https://registry.opentofu.org/v1/providers/hashicorp/kubernetes - 404

But

https://registry.opentofu.org/v1/providers/hashicorp/kubernetes/versions - works

[quotidian-ennui]

I tested with the latest 0.71.0 which works

  tf-lock-k8s:
    name: Update hashicorp/kubernetes lockfile
    kind: terraform/lock
    sourceid: terraform-provider-k8s
    spec:
      file: terraform/.terraform.lock.hcl
      provider: registry.opentofu.org/hashicorp/kubernetes
      platforms:
        - linux_amd64
        - windows_amd64
        - darwin_amd64
        - darwin_arm64
    dependson:
      - tf-provider-k8s
    dependsonchange: true

[kvendingoldo]

At this moment if you need to have a support of Terraform as well as OpenTofu (and Terragrunt :) ) in one tool you can use https://github.com/tofuutils/tenv which my team wrote some months ago. A lot of users switched to that tool to unify version management in the world of Terraform.

You're welcome to open any issues or contribute to tenv.

[loispostula]

@mcwarman Have you tested this issue since #2372 has been released?

[quotidian-ennui]

@loispostula -> I am running this with v0.88.1 which includes your PR; whether or not that actually answers the question you're asking is of course is another question ;)

It's been running quite happily.

# {{ $opentofu := eq (env "TF_BINARY" | default "tofu") "tofu" }}
name: github-terraform-provider
version: v0.7.1

sources:
  github-terraform-provider:
    name: Github release integrations/github
    kind: githubrelease
    spec:
      owner: integrations
      repository: terraform-provider-github
      token: '{{ requiredEnv "GITHUB_TOKEN" }}'
      versionfilter:
        kind: semver
        pattern: '*'
    transformers:
      - trimprefix: "v"

targets:
  github-lock-github:
    name: Update .terraform.lock.hcl in github/ to {{ source "github-terraform-provider" }}
    kind: terraform/lock
    sourceid: github-terraform-provider
    spec:
      file: github/.terraform.lock.hcl
      provider: "{{ if $opentofu }}registry.opentofu.org/{{ end }}integrations/github"
      platforms:
        - linux_amd64
        - windows_amd64
        - darwin_amd64
        - darwin_arm64
    dependson:
      - github-provider-github
    dependsonchange: true

  github-provider-github:
    name: Update integrations/github in github/ to {{ source "github-terraform-provider" }}
    kind: terraform/provider
    sourceid: github-terraform-provider
    spec:
      file: github/versions.tf
      provider: github