Python: .NET: [Bug]: ContainsExpected reports missing expected output as a model failure #9019

Summary

EvalChecks.ContainsExpected() returns a failed check when EvalItem.ExpectedOutput is null or empty, even though the check cannot be applied. This turns a missing test fixture into a failed agent evaluation and is inconsistent with the built-in expected-tool-call check, which treats absent expectations as non-failing.

Version and evidence

Reproduced from main at commit e9857bd75a21843803431a2e7dfa41ee9b017ac7 on 2026-10-03.

• ExpectedOutput is nullable in dotnet/src/Microsoft.Agents.AI/Evaluation/EvalItem.cs:108-114.
• ContainsExpected maps null or empty ExpectedOutput to EvalCheckResult(false, "ExpectedOutput is not set; check cannot be applied.", ...) in dotnet/src/Microsoft.Agents.AI/Evaluation/EvalChecks.cs:219-243.
• LocalEvaluator converts any false check result into a failed BooleanMetric in dotnet/src/Microsoft.Agents.AI/Evaluation/LocalEvaluator.cs:43-58.
• By contrast, ToolCallArgsMatch explicitly passes when ExpectedToolCalls is null or empty in EvalChecks.cs:126-146.
• The existing test records the current failure behavior in dotnet/tests/Microsoft.Agents.AI.UnitTests/EvaluationTests.cs:1510-1518.

Package version: source build at the SHA above. The reproduction was compiled for net10.0 and run with .NET SDK 10.0.400; the repository pins 10.0.401.

Minimal reproduction

Proposed xUnit regression test in dotnet/tests/Microsoft.Agents.AI.UnitTests/EvaluationDefectRegressionTests.cs:

[Fact]
public void ContainsExpected_WithoutExpectedOutput_ThrowsConfigurationError()
{
    // Arrange
    EvalCheck check = EvalChecks.ContainsExpected();
    EvalItem item = new(query: "question", response: "answer");

    // Act & Assert: a missing fixture is a configuration problem, not a model failure.
    Assert.Throws<InvalidOperationException>(() => check(item));
}

Run with:

dotnet run --project dotnet/tests/Microsoft.Agents.AI.UnitTests/Microsoft.Agents.AI.UnitTests.csproj `
  -f net10.0 --no-restore -- `
  --filter-query "/*/Microsoft.Agents.AI.UnitTests/EvaluationDefectRegressionTests/ContainsExpected_WithoutExpectedOutput_ThrowsConfigurationError"

The test fails with Assert.Throws() Failure: No exception was thrown; the returned result has Passed == false and reason ExpectedOutput is not set; check cannot be applied.

Actual behavior

A missing fixture produces an ordinary failed check, and LocalEvaluator reports it as an agent evaluation failure (EvalChecks.cs:224-242, LocalEvaluator.cs:43-58). ToolCallArgsMatch uses the opposite missing-input policy and returns a passing check (EvalChecks.cs:138-146).

Expected behavior

Missing ExpectedOutput should be surfaced as test/evaluator configuration, not as evidence that the agent failed. The regression test expresses the recommended behavior: throw a clear InvalidOperationException before an evaluation metric is produced.

Impact

Any test suite that conditionally supplies expected outputs can produce false model failures when one fixture is omitted. CI output points maintainers toward the agent response even though the actionable defect is in the test data; this is especially confusing because the reason itself says that the check could not be applied (EvalChecks.cs:228-230).

Python parity

Python has no built-in contains_expected check. Function-evaluator injection converts missing expected_output to "" in python/packages/core/agent_framework/_evaluation.py:1253-1301, and its focused test asserts that conversion in python/packages/core/tests/core/test_local_eval.py:127-149. Python's expected-tool-call checks pass when no expectations are supplied in _evaluation.py:1141-1164 and _evaluation.py:1183-1208, so Python does not have the same built-in failure behavior.

Duplicate search

No open or closed issue or PR matched ContainsExpected. A broader ExpectedOutput "not set" search only surfaced unrelated sample PR #5823, which is not a duplicate.

Fix options

1. Throw a configuration exception (recommended). Throw InvalidOperationException when ExpectedOutput is null or empty. This is an observable behavior change for invalid/missing fixtures, but it changes no signature and requires no PublicAPI.Shipped.txt or PublicAPI.Unshipped.txt update; ContainsExpected is already shipped in dotnet/src/Microsoft.Agents.AI/PublicAPI/net10.0/PublicAPI.Shipped.txt:946.
2. Skip/pass when no expectation is supplied. Return a passing result, matching ToolCallArgsMatch. This is also a behavior change with no public API baseline change and is convenient for mixed datasets, but it can silently hide an accidentally missing fixture.
3. Represent applicability separately. Add an applicability/configuration state to EvalCheckResult and teach LocalEvaluator how to aggregate it. This gives the most precise diagnostics, but EvalCheckResult is a shipped public record in PublicAPI.Shipped.txt:295-305; changing its primary constructor/deconstructor would be a public API break, while an additive property would require PublicAPI.Unshipped.txt entries and more aggregation design.

I recommend option 1 because it fails fast on the real problem without silently accepting incomplete fixtures or changing public API. I would be happy to submit a PR with the regression test and a documented missing-input policy for the built-in checks.