testing: include file-level errors in JUnit skipped elements

[vietcgi]

Fixes #37801

When test files have file-level error diagnostics (e.g., invalid variable references), tests are skipped but the JUnit XML output doesn't include the error details. This causes CI/CD pipelines to miss failures.

This PR adds file-level error diagnostics to the skipped element body, ensuring errors are visible in JUnit XML reports.

Changes:

• Modified skipDetails to check for file-level error diagnostics
• Added diagnostic details to skipped test output when file errors exist
• Added test coverage for file-level error scenarios

Impact:
Pipelines checking JUnit XML for errors will now correctly detect file-level validation failures.

[hashicorp-cla-app]

All committers have signed the CLA.

[SarahFrench]

Nice!

[SarahFrench]

Hi @vietcgi, thank you for opening this PR and including test coverage of the change 🙌🏻

I was looking at this resource on GitHub the describes the JUnit XML specification and thought that it could be useful to surface the file-level error via a <system-err> element nested under <testsuite>, as the test suite concept maps to a single test file. This could be in addition to logging the error in the <skipped> element, or replace that approach.

Do you have any preferences either way? I haven't used JUnit XML output in production myself so I'm unfamiliar with how the XML is consumed by commercial tools and whether one approach fits better with those tools versus the other approach, or not. One benefit I can see is that users could grep for <system-err> being present, similar to the example shared here, instead of needing to inspect the error message contents.

Just an FYI, implementing use of <system-err> would be a bit more complex but could resemble the existing approach used with enc.EncodeElement and the testSuite struct.

[vietcgi]

Hi @SarahFrench

Good call on the <system-err> suggestion, that's cleaner.

You're right that file-level errors belong at the suite level, not testcase level. When the file itself is broken, it's not that the tests failed, it's that they couldn't even run.

I'll move the diagnostic details to a suite-level <system-err> element (after all testcases) and keep the skipped elements simple. That way CI tools can just grep for <system-err> instead of parsing message content.

The updated structure will look like:

<testsuite name="file.tftest.hcl" tests="1" skipped="1" failures="0" errors="0">
  <testcase name="my_test" classname="file.tftest.hcl">
    <skipped message="Skipped due to file-level errors"></skipped>
  </testcase>
  <system-err><![CDATA[
Error: Invalid reference
...
]]></system-err>
</testsuite>

Does that direction work for you?

[SarahFrench]

The updated structure will look like:

<testsuite name="file.tftest.hcl" tests="1" skipped="1" failures="0" errors="0">
  <testcase name="my_test" classname="file.tftest.hcl">
    <skipped message="Skipped due to file-level errors"></skipped>
  </testcase>
  <system-err><![CDATA[
Error: Invalid reference
...
]]></system-err>
</testsuite>

Does that direction work for you?

That looks good to me, thanks for being open to the feedback!

[vietcgi]

it's good now @SarahFrench :)