Provider prevents unit testing client code

[arlobelshee]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and Azure DevOps Provider) Version

Terraform: 1.1.2
Provider: 0.1.8

Affected Resource(s)

• all

Terraform Configuration Files

provider "azuredevops" {
  org_service_url = [valid org URL]
}

terraform {
  required_providers {
    azuredevops = {
      source = "microsoft/azuredevops"
      version = ">=0.1.8"
    }
  }
}

Expected Behavior

It should have output a plan with 0 changes.

Actual Behavior

It crashed, with error message:

Error: TF400813: The user '' is not authorized to access this resource.

Steps to Reproduce

1. Set $env:AZDO_PERSONAL_ACCESS_TOKEN = 'Fake token'
2. terraform init
3. terraform plan -refresh=false

Important Factoids

Org does not allow anonymous access (and shouldn't).

[arlobelshee]

This is critical for unit testing. My unit tests do not run against any actual resources. in fact, I would like them to fail if they do attempt to access a real resource.

The tests run by generating various plan output files, then making assertions. Each plan output is generated from a known starting state (stored as part of the test base), then applying a particular set of resources. I verify it does the right thing. This allows me to test, eg, the logic in my modules that does different things for production vs dev.

All other providers work fine. They only attempt to log in when they actually need to do an API call. So I can give them invalid credentials and they will only use them (and crash) if my test has a bug and attempts to actually reference a real resource (read or modify).

This provider always attempts to log in, even if it doesn't need to reference a resource. That prevents unit testing.

Also, my CI server cannot have a PAT, not even with read access. That would violate part of our security model. So I cannot simply create a "valid set of credentials" and then use them.

[xuzhang3]

@arlobelshee Can you make sure the test script does not contain any ADO resource resource/data source configures? If the test script only have the provider configuration, the fake PAT should work without any error.

[arlobelshee]

The problem is that I need to perform unit testing. So I'm testing some module, for example:

Module's main:

resource "azuredevops_project" "project" {
  # Content and settings...
}

resource "azuredevops_git_repository" "primary" {
  project_id = azuredevops_project.project.id
  name       = azuredevops_project.project.name
  initialization {
    init_type = "Clean"
  }
}

resource "azuredevops_git_repository" "docs" {
  project_id = azuredevops_project.project.id
  name       = "Docs"
  initialization {
    init_type = "Clean"
  }
}

# ...and so on. Much more.

Then there is a feature definition like:

Feature Project exists and has required repositories

	@noskip
	Scenario: Ensure the project exists
		Given I have azuredevops_project defined

	@noskip
	Scenario: Ensure the primary repo exists
		Given I have azuredevops_project defined
		When its name is TestProject

	@noskip
	Scenario: Ensure the documentation repo exists
		Given I have azuredevops_project defined
		When its name is docs

The set of features have a terraform main like:

module "test_subject" {
  src = "../"  # This references the module's directory.
  project_name = "TestProject"
}

I then run this with:

terraform init
terraform plan -refresh=false -out test.tfplan
terraform-compliance -f . -p test.tfplan

This works because I have an initial state set for the test (a committed .terraform folder with my starting state). The -refresh=false means that terraform won't try to query the APIs for any real set of resources to make sure that the state is up to date. It'll just use the state I have. And because I'm just generating the plan, terraform won't make any changes. Thus, it never calls any API method for a resource.

This works for all other providers, because they don't attempt to log in until the first actual API call. However, this provider breaks because it always attempts to log in during init, even if I will never make an API call.

[arlobelshee]

To be clear, one of my explicit goals is that the unit tests don't need any credentials at all - not even a fake. If absolutely necessary, I'd be possibly OK with setting things like:

OrganizationUrl = 'https://fake.example.com/fake'
Pat = 'Fake pat that will never be used'

But I'd rather not do even that - I don't need to do anything like that for any other provider that I use.

I am disallowed (by security design) from using a PAT that could actually log in successfully, no matter how small its permission set.

[arlobelshee]

Any update, @xuzhang3?

[xuzhang3]

@arlobelshee I see, this provider will try to create the client connection and verify the resource even you set -refresh=false,