Calamari TerraformCliExecutor Broken for Remote Backend Workspace Prefix

[nfb-transact]

Severity

No response

Version

main

Latest Version

None

What happened?

The code in InitializePlugins() calling terraform init before InitializeWorkspace() is fundamentally broken for remote backends with prefix. The workspace must be established before or during init, not after.

https://github.com/OctopusDeploy/Calamari/blob/main/source/Calamari.Terraform/TerraformCliExecutor.cs#L86C13-L86C30

            InitializeTerraformEnvironmentVariables();
            Version = GetVersion();
            InitializePlugins(); // here it does init without workspace selected
            InitializeWorkspace();

Am I just missing something or is this a fundamental bug in TerraformCliExecutor?

Reproduction

Use built-in OctopusDeploy Terraform steps with a Terraform configuration such as:

terraform {
  backend "remote" {
    hostname     = "myhost.jfrog.io"
    organization = "myorg"
    workspaces {
      prefix = "myprefix"
    }
  }
}

Error and Stacktrace

"terraform" version --json 
{ 
  "terraform_version": "1.8.5", 
  "platform": "linux_amd64", 
  "provider_selections": { 
    "registry.terraform.io/hashicorp/aws": "6.32.1", 
    "registry.terraform.io/hashicorp/helm": "2.17.0", 
    "registry.terraform.io/hashicorp/kubernetes": "2.38.0", 
    "registry.terraform.io/hashicorp/time": "0.13.1", 
    "registry.terraform.io/hashicorp/tls": "4.2.1" 
  }, 
  "terraform_outdated": true 
} 
"terraform" init -no-color  
Initializing the backend... 
The currently selected workspace (alpha-eks) does not exist. 
  This is expected behavior when the selected workspace did not have an 
  existing non-empty state. Please enter a number to select a workspace: 
  1. dev 
  2. local 
  3. local-eks 
  Enter a value:  
Successfully configured the backend "remote"! Terraform will automatically 
use this backend unless the backend configuration changes. 
Initializing modules... 
- eks in modules 
Downloading registry.terraform.io/terraform-aws-modules/kms/aws 4.0.0 for eks.kms... 
- eks.kms in .terraform/modules/eks.kms 
- grafana in modules/managed-grafana 
- prometheus in modules/managed-prometheus 
Downloading registry.terraform.io/terraform-aws-modules/vpc/aws 6.6.0 for vpc... 
- vpc in .terraform/modules/vpc 
- vpc_peering_db_compute in modules/vpc-peering 
- vpc_peering_legacy_compute in modules/vpc-peering 
Error: Failed to select workspace: EOF 
Running rollback behaviours... 
The following command: "terraform" init -no-color  
With the working directory of: /home/octoworker/octopus/default/Work/G51Peun5SU6PpRd0P4OnXw/staging/terraform 
Failed with exit code: 1 
Error: Failed to select workspace: EOF 
Calamari.Common.Features.Processes.CommandLineException 
   at Calamari.Common.Features.Processes.CommandResult.VerifySuccess() in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Common\Features\Processes\CommandResult.cs:line 28 
   at Calamari.Terraform.TerraformCliExecutor.VerifySuccess(CommandResult commandResult, Predicate`1 isSuccess) in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Terraform\TerraformCliExecutor.cs:line 151 
   at Calamari.Terraform.TerraformCliExecutor.VerifySuccess(CommandResult commandResult) in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Terraform\TerraformCliExecutor.cs:line 156 
   at Calamari.Terraform.TerraformCliExecutor.ExecuteCommandAndVerifySuccess(String[] arguments, String& result, Boolean outputToCalamariConsole) in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Terraform\TerraformCliExecutor.cs:line 180 
   at Calamari.Terraform.TerraformCliExecutor.InitializePlugins() in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Terraform\TerraformCliExecutor.cs:line 220 
   at Calamari.Terraform.TerraformCliExecutor..ctor(ILog log, ICalamariFileSystem fileSystem, ICommandLineRunner commandLineRunner, RunningDeployment deployment, Dictionary`2 environmentVariables) in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Terraform\TerraformCliExecutor.cs:line 85 
   at Calamari.Terraform.Behaviours.PlanBehaviour.Execute(RunningDeployment deployment, Dictionary`2 environmentVariables) in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Terraform\Behaviours\PlanBehaviour.cs:line 47 
   at Calamari.Terraform.Behaviours.TerraformDeployBehaviour.Execute(RunningDeployment context) in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Terraform\Behaviours\TerraformDeployBehaviour.cs:line 59 
   at Calamari.Common.Plumbing.Pipeline.PipelineCommand.ExecuteBehaviour(RunningDeployment context, IBehaviour behaviour) in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Common\Plumbing\Pipeline\PipelineCommand.cs:line 164 
   at Calamari.Common.Plumbing.Pipeline.PipelineCommand.Execute(ILifetimeScope lifetimeScope, IVariables variables) in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Common\Plumbing\Pipeline\PipelineCommand.cs:line 60 
   at Calamari.Common.Plumbing.Pipeline.PipelineCommand.Execute(ILifetimeScope lifetimeScope, IVariables variables) in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Common\Plumbing\Pipeline\PipelineCommand.cs:line 84 
   at Calamari.Common.CalamariFlavourProgramAsync.Run(String[] args) in C:\BuildAgent\work\19cee70d4897f05b\source\Calamari.Common\CalamariFlavourProgramAsync.cs:line 148 
The remote script failed with exit code 100 
The action Terraform Plan on a Worker failed

More Information

No response

Workaround

I could use Octopus variable substitution to dynamically insert a workspace name I believe, but I would rather not encode Octopus specifics into my Terraform config.