Errors in test are skipped vs failed since 1.13.0

[lapkritinis]

Terraform Version

Terraform v1.13.4
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v6.17.0
+ provider registry.terraform.io/hashicorp/http v3.5.0
+ provider registry.terraform.io/hashicorp/local v2.5.3

Terraform Configuration Files

a.tf

data "aws_caller_identity" "current" {}

resource "aws_s3_bucket" "this" {}

a.tftest.hcl

variables {
   account_id = data.aws_caller_identity.current.account_id
}

run "test_s3_bucket_name" {
  command = apply

  assert {
    condition     = aws_s3_bucket.this.bucket != var.account_id
    error_message = "S3 Bucket name not matching account id"
  }
}

Debug Output

Probably not relevant

Expected Behavior

1.12.0 fails test:

terraform test -verbose -junit-xml=test.xml
a.tftest.hcl... in progress
  run "test_s3_bucket_name"... fail
╷
│ Error: Invalid reference
│ 
│   on a.tftest.hcl line 2, in variables:
│    2:    account_id = data.aws_caller_identity.current.account_id
│ 
│ You can only reference global variables within the test file variables block.
╵
a.tftest.hcl... tearing down
a.tftest.hcl... fail

Failure! 0 passed, 1 failed

Generated junit-xml contains error as expected:

<?xml version="1.0" encoding="UTF-8"?><testsuites>
  <testsuite name="a.tftest.hcl" tests="1" skipped="0" failures="0" errors="1">
    <testcase name="test_s3_bucket_name" classname="a.tftest.hcl" time="1.229734" timestamp="2025-10-22T14:08:07Z">
      <error message="Encountered an error"><![CDATA[
Error: Invalid reference

  on a.tftest.hcl line 2, in variables:
   2:    account_id = data.aws_caller_identity.current.account_id

You can only reference global variables within the test file variables block.
]]></error>
    </testcase>
  </testsuite>
</testsuites

Actual Behavior

But since version 1.13.0 it behaves differently. Tested on 1.13.4

terraform test -verbose -junit-xml=test.xml
a.tftest.hcl... in progress
  run "test_s3_bucket_name"... skip
a.tftest.hcl... tearing down
a.tftest.hcl... fail
╷
│ Error: Invalid reference
│ 
│   on a.tftest.hcl line 2, in variables:
│    2:    account_id = data.aws_caller_identity.current.account_id
│ 
│ You can only reference run blocks and variables from within Terraform Test files.
╵

Failure! 0 passed, 0 failed, 1 skipped.

unit xml contains no error:

<?xml version="1.0" encoding="UTF-8"?><testsuites>
  <testsuite name="a.tftest.hcl" tests="1" skipped="1" failures="0" errors="0">
    <testcase name="test_s3_bucket_name" classname="a.tftest.hcl">
      <skipped></skipped>
    </testcase>
  </testsuite>
</testsuites

Problem is with automated test in pipelines this can be easily missed as failure. I would expect it to behave like prior version 1.12.0 and before.

Steps to Reproduce

terraform test with my provided test "files"

Additional Context

No response

References

No response

Generative AI / LLM assisted development?

No response

[liamcervante]

Hi @lapkritinis, thanks for filing this! The change here is to reflect that the error isn't with the specific test case but instead with the file in general - the problematic variable exists outside the scope of any test cases and so the tests are actually skipped as a result of the data they need not being available.

However, this does seem to have been missed from the Junit output - it should still include the diagnostic in there somewhere so we will look into fixing this to make sure the exact error is surfaced somehow.

[lapkritinis]

Thanks @liamcervante, that would solve my issue as we are checking Junit XML in pipeline for errors or failures:

if grep -q '</failure>' ./test.xml || grep -q '</error>' ./test.xml; then.... fail pipeline