tfversion: Consider Ignoring Terraform Prerelease Version Information in Certain Checks #303
Description
terraform-plugin-testing version
v1.7.0
Use cases
When Terraform introduces new functionality, it will appear in prereleases of the minor version before general availability. This is true for provider-facing features, the Terraform built-in provider features, and purely configuration-based features.
The terraform-plugin-testing Go module supports automatic test skipping and test failures based on the Terraform version via the TestCase.TerraformVersionChecks field and the various built-in check implementations in the tfversion package.
In this example, tests will automatically be skipped if the Terraform version is below 1.8.0 (e.g. 1.7.x, 1.6.x, etc.):
TerraformVersionChecks: []tfversion.TerraformVersionCheck{
tfversion.SkipBelow(tfversion.Version1_8_0),
},Especially when it comes to provider-facing features though, there is a time period during these Terraform prereleases where provider developers may want to implement the new features with acceptance testing, but also ensure that testing only runs on the upcoming version or later. For example, many of the HashiCorp-owned "utility" providers explicitly run their testing against every minor version of Terraform supported to detect potential regressions or version-specific quirks so these version checks working are important to not raise false positives or unexpectedly pass testing by skipping it.
Given the above check, running the testing while using Terraform 1.8.0-beta1 will skip the test:
$ TF_ACC_TERRAFORM_PATH=/Users/bflad/bin/terraform1.8.0-beta1 TF_ACC=1 go test -count=1 -run='TestRFC3339Parse_UTC' -v ./internal/provider
=== RUN TestRFC3339Parse_UTC
function_rfc3339_parse_test.go:18: Terraform CLI version 1.8.0-beta1 is below minimum version 1.8.0: skipping test
--- SKIP: TestRFC3339Parse_UTC (1.18s)For certain version checks, it may be desirable to "allow"/"ignore" prerelease versions given that the intention of these checks is to detect versions based on the core version information.
Attempted solutions
As a workaround, it is possible to inject prerelease information into tfversion package checks:
TerraformVersionChecks: []tfversion.TerraformVersionCheck{
// TODO: Replace with the stable v1.8.0 release when available
tfversion.SkipBelow(version.Must(version.NewVersion("1.8.0-beta1"))),
},This requires updates until the final release occurs.
Proposal
In at least SkipBelow and RequireAbove checks, update the logic to ignore prerelease information when performing the comparison. github.com/hashicorp/go-version supports calling (*version.Version).Core() for example, which could be used against the Terraform version. If compatibility concerns are absolutely important in this case, then the Core() call could potentially be ignored if (*version.Version).Prerelease() is non-empty for the version given to the check.
Another idea could be to introduce separate checks with the slightly different prerelease behavior, although it seems like explicit Terraform prerelease handling should be treated as exceptional or less common use given how Terraform is versioned in practice.