test: group related assertions with Assert.Multiple - #25
Conversation
Wrap consecutive TUnit assertions in using (Assert.Multiple()) so a failing assertion no longer hides the remaining checks in the same test, making failures easier to diagnose in one run.
WalkthroughThe pull request updates integration and unit tests to wrap related assertions in ChangesGrouped test assertions
Estimated code review effort: 3 (Moderate) | ~25 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #25 +/- ##
=======================================
Coverage 95.14% 95.14%
=======================================
Files 70 70
Lines 4659 4659
Branches 1109 1109
=======================================
Hits 4433 4433
Misses 91 91
Partials 135 135 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
samtrion
added
state:ready for merge
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MSTestSurfaceAnalyzerTests.cs`:
- Around line 182-189: Keep the dependent diagnostics[0] assertions behind a
diagnostics.Length == 1 gate so Assert.Multiple() cannot evaluate them after the
count assertion fails. Apply this pattern at
tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MSTestSurfaceAnalyzerTests.cs:182-189,
MixedFrameworkTestSurfaceTests.cs:580-584 and :715-719,
NUnitTestSurfaceAnalyzerTests.cs:240-247,
TUnitTestSurfaceAnalyzerTests.cs:368-375, :427-437, and :803-807, and
XunitV2TestSurfaceAnalyzerTests.cs:225-232 and :371-380; retain the count
assertion and only run index-based location/message checks when exactly one
diagnostic exists.
In
`@tests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutationOperatorGuardTests.cs`:
- Around line 194-199: In MutationOperatorGuardTests.cs:194-199, move the
mutations.Length assertion outside Assert.Multiple before indexing mutations,
then keep only safe grouped assertions. Apply the same dependent-expression
fixes in TUnitTestFrameworkProbeTests.cs:259-263 and
TestMethodDiscoveryTests.cs:259-264 and 327-334: perform collection-size checks
before indexing, and ensure nullable values are asserted non-null before
accessing their members; sites corrected by the shared assertion pattern require
the corresponding direct updates.
In
`@tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparerMutatorTests.cs`:
- Around line 423-427: Make dependent mutation lookups non-throwing in the
listed blocks using MutationTo within StringComparerMutatorTests.cs at lines
423-427, 436-440, 449-453, 466-470, and 558-562, and
StringComparisonMutatorTests.cs at lines 432-436, 449-453, 462-466, 479-483, and
571-575: retrieve the expected mutation nullable, then guard Rewrite so missing
mutations become collected assertion failures rather than precondition
exceptions while preserving the existing Assert.Multiple assertions.
In
`@tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs`:
- Around line 103-110: The collection-dependent assertions must not access
elements or derive substrings while their count, emptiness, or index
preconditions are only asserted inside Assert.Multiple(); guard those accesses
or use non-indexing predicates so assertion failures remain diagnostic. Apply
this to StringLiteralMutatorTests.cs ranges 103-110, 118-127, 134-139, 152-159,
173-180, 206-213, 221-228, and 327-333; UnaryOperatorMutatorTests.cs 402-408;
MutantCompilerTests.cs 285-291; RegexPatternTokenizerAdversarialTests.cs 463-469
and 918-925; and RegexPatternTokenizerTests.cs 546-553.
In
`@tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs`:
- Around line 421-431: Update the Assert.Multiple blocks in
RegexPatternLocatorTests.cs at
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs:421-431,
438-445, 461-468, 475-481, 494-503, 510-514, 526-532, 547-553, 565-571, 591-596,
618-625, 659-666, 687-692, 702-708, 838-843, 951-960, 972-979, 989-995,
1005-1009, 1042-1048, 1086-1094, and 1107-1113 to use assertion shapes that
remain independently evaluable when earlier checks fail; avoid
nullable-forgiving access, Nullable<T>.Value, or Parent/property dereferences
that can throw before later assertions report their intended diagnostics.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 5f14a249-e1a0-4623-b822-42f145adcbd7
📒 Files selected for processing (68)
tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/CultureMutationTests.cstests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MSTestSurfaceAnalyzerTests.cstests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MixedFrameworkTestSurfaceTests.cstests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MutationCoverageAnalyzerTests.cstests/NetEvolve.FrameShift.Tests.Integration/Analyzers/NUnitTestSurfaceAnalyzerTests.cstests/NetEvolve.FrameShift.Tests.Integration/Analyzers/TUnitTestSurfaceAnalyzerTests.cstests/NetEvolve.FrameShift.Tests.Integration/Analyzers/TestMarkerRecognitionTests.cstests/NetEvolve.FrameShift.Tests.Integration/Analyzers/XunitV2TestSurfaceAnalyzerTests.cstests/NetEvolve.FrameShift.Tests.Integration/Analyzers/XunitV3TestSurfaceAnalyzerTests.cstests/NetEvolve.FrameShift.Tests.Integration/Generation/TestSurfaceManifestGeneratorTests.cstests/NetEvolve.FrameShift.Tests.Integration/MutationAnalysisRoundTripTests.cstests/NetEvolve.FrameShift.Tests.Unit/Configuration/FrameShiftOptionsTests.cstests/NetEvolve.FrameShift.Tests.Unit/Equivalence/EquivalenceClassifierBranchTests.cstests/NetEvolve.FrameShift.Tests.Unit/Equivalence/EquivalenceClassifierFoldingOperatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Equivalence/EquivalenceClassifierFoldingTests.cstests/NetEvolve.FrameShift.Tests.Unit/Equivalence/EquivalenceClassifierTests.cstests/NetEvolve.FrameShift.Tests.Unit/Equivalence/EquivalenceVerdictTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutantCompilerTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutantGeneratorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutationOperatorGuardTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutationOperatorRegistryTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutationTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/ArithmeticAssignmentMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/ArithmeticOperatorMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/BitwiseOperatorMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/BooleanLiteralMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/CaseConversionMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/ConditionalExpressionMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/CultureInfoMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/EqualityOperatorMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/FormatProviderArgumentMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/IncrementDecrementMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/LogicalNegationMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/LogicalOperatorMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/NullCoalescingMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/NullableBooleanLiteralMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/NumericLiteralMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/RegexOptionsMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/RelationalOperatorMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparerMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparisonMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/UnaryOperatorMutatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerAdversarialTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerTests.cstests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternValidityTests.cstests/NetEvolve.FrameShift.Tests.Unit/Reachability/ReachabilityClosureMemberTests.cstests/NetEvolve.FrameShift.Tests.Unit/Reachability/ReachabilityClosurePolymorphismTests.cstests/NetEvolve.FrameShift.Tests.Unit/Reachability/ReachabilityClosureTests.cstests/NetEvolve.FrameShift.Tests.Unit/Reachability/ReachableSymbolSetTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/MSTestFrameworkProbeTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/MSTestMethodRecognizerTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/NUnitTestFrameworkProbeTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/NUnitTestMethodRecognizerTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TUnitTestFrameworkProbeTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TUnitTestMethodRecognizerTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TestCaseCountTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TestMethodDiscoveryTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TestSurfaceCollectorMemberTraversalTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TestSurfaceCollectorTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TestSurfaceManifestReaderTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TestSurfaceManifestTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TestSurfaceManifestWriterTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/XunitV2TestFrameworkProbeTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/XunitV2TestMethodRecognizerTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/XunitV3TestFrameworkProbeTests.cstests/NetEvolve.FrameShift.Tests.Unit/TestSurface/XunitV3TestMethodRecognizerTests.cs
| using (Assert.Multiple()) | ||
| { | ||
| _ = await Assert.That(diagnostics.Length).IsEqualTo(1); | ||
| _ = await Assert.That(diagnostics[0].Location.SourceSpan).IsEqualTo(identifier.Span); | ||
| _ = await Assert | ||
| .That(GetMessage(diagnostics[0]).Contains(LocalOnlyTestName, StringComparison.Ordinal)) | ||
| .IsTrue(); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the referenced test files and inspect asserted relevant snippets.
for f in \
tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MSTestSurfaceAnalyzerTests.cs \
tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MixedFrameworkTestSurfaceTests.cs \
tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/NUnitTestSurfaceAnalyzerTests.cs \
tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/TUnitTestSurfaceAnalyzerTests.cs \
tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/XunitV2TestSurfaceAnalyzerTests.cs
do
echo "== $f =="
if [ -f "$f" ]; then
rg -n --only-matching 'Assert\.Multiple\(\)|diagnostics\[[^]]+\]\.Location|otherDiagnostics\[[^]]+\]\.Location|\.Count\(\)\.IsEqualTo\(1\)|Contains\([^)]+\|StringComparison\.Ordinal\)' "$f"
echo "--- relevant sections ---"
rg -n -C 4 'Assert\.Multiple\(\)' "$f" | sed -n '1,220p'
else
echo "MISSING"
fi
done
echo "== TUnit Assert.Multiple docs references in repository/config?"
rg -n 'Assert\.Multiple|Multiple\(\)|tunit\.dev|Combining assertions' . -g '*.cs' -g '*.md' -g '*.txt' -g '*.yml' -g '*.yaml' | sed -n '1,180p'Repository: dailydevops/frameshift
Length of output: 35883
🌐 Web query:
TUnit Assert.Multiple continue after assertion failed documentation
💡 Result:
In TUnit, Assert.Multiple is specifically designed to allow multiple assertions to execute within a test without stopping at the first failure [1]. When assertions are wrapped in a using (Assert.Multiple) block, any failures encountered are accumulated and reported together at the end of the block, rather than throwing an exception immediately upon the first failure [2][1]. Key behaviors and documentation points: - Purpose: It allows you to group related assertions so that all failures are captured and reported, which is useful for diagnosing multiple issues in a single test run [1]. - Implementation: You use it with a using statement. All assertions inside this block that are awaited will be tracked by the scope [1]. - Assertion Execution: It is critical that all assertions within the scope are awaited (using await Assert.That(...)) [3][1]. Unawaited assertions will not execute, and the test may pass incorrectly [1]. - Behavior with Assert.Fail: Historically, there have been nuances regarding whether all methods behave consistently within these blocks. While standard Assert.That calls accumulate correctly [2][1], you should ensure that the assertion methods you use are designed to participate in the TUnit assertion pipeline [2]. If you find that your test is still stopping prematurely, ensure that you are using standard TUnit assertions and that every assertion within the using block is properly awaited [1].
Citations:
- 1: https://tunit.dev/docs/assertions/getting-started/
- 2: fix: IsNotNull assertion thomhurst/TUnit#3634
- 3: https://tunit.dev/docs/getting-started/writing-your-first-test/
Keep dependent diagnostic assertions behind a count gate.
Assert.Multiple() aggregates failures instead of stopping on the first failed assertion. If the diagnostics count assertion fails, the following diagnostics[0] assertions still execute and can throw with a less useful exception. Skip index-based assertions when the count is not the expected single value.
Proposed pattern
using (Assert.Multiple())
{
_ = await Assert.That(diagnostics).Count().IsEqualTo(1);
- _ = await Assert.That(diagnostics[0].Location.SourceSpan).IsEqualTo(identifier.Span);
- _ = await Assert.That(GetMessage(diagnostics[0]).Contains(name, StringComparison.Ordinal)).IsTrue();
+ if (diagnostics.Length == 1)
+ {
+ _ = await Assert.That(diagnostics[0].Location.SourceSpan).IsEqualTo(identifier.Span);
+ _ = await Assert.That(GetMessage(diagnostics[0]).Contains(name, StringComparison.Ordinal)).IsTrue();
+ }
}📍 Affects 5 files
tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MSTestSurfaceAnalyzerTests.cs#L182-L189(this comment)tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MixedFrameworkTestSurfaceTests.cs#L580-L584tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MixedFrameworkTestSurfaceTests.cs#L715-L719tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/NUnitTestSurfaceAnalyzerTests.cs#L240-L247tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/TUnitTestSurfaceAnalyzerTests.cs#L368-L375tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/TUnitTestSurfaceAnalyzerTests.cs#L427-L437tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/TUnitTestSurfaceAnalyzerTests.cs#L803-L807tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/XunitV2TestSurfaceAnalyzerTests.cs#L225-L232tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/XunitV2TestSurfaceAnalyzerTests.cs#L371-L380
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MSTestSurfaceAnalyzerTests.cs`
around lines 182 - 189, Keep the dependent diagnostics[0] assertions behind a
diagnostics.Length == 1 gate so Assert.Multiple() cannot evaluate them after the
count assertion fails. Apply this pattern at
tests/NetEvolve.FrameShift.Tests.Integration/Analyzers/MSTestSurfaceAnalyzerTests.cs:182-189,
MixedFrameworkTestSurfaceTests.cs:580-584 and :715-719,
NUnitTestSurfaceAnalyzerTests.cs:240-247,
TUnitTestSurfaceAnalyzerTests.cs:368-375, :427-437, and :803-807, and
XunitV2TestSurfaceAnalyzerTests.cs:225-232 and :371-380; retain the count
assertion and only run index-based location/message checks when exactly one
diagnostic exists.
| using (Assert.Multiple()) | ||
| { | ||
| _ = await Assert.That(mutator.CoreInvocations).IsEqualTo(1); | ||
| _ = await Assert.That(mutations.Length).IsEqualTo(1); | ||
| _ = await Assert.That(mutations[0].OperatorId).IsEqualTo("probe.identity"); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Repo files (matching test files):\n'
git ls-files | rg 'MutationOperatorGuardTests\.cs$|TUnitTestFrameworkProbeTests\.cs$|TestMethodDiscoveryTests\.cs$' || true
printf '\nRelevant usages of Assert.Multiple in target files:\n'
for f in \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutationOperatorGuardTests.cs \
tests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TUnitTestFrameworkProbeTests.cs \
tests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TestMethodDiscoveryTests.cs
do
if [ -f "$f" ]; then
echo "--- $f ($(wc -l < "$f") lines) ---"
rg -n "Assert\.Multiple|mutations|resolved|found" "$f" -C 2 || true
else
echo "missing: $f"
fi
done
printf '\nCandidate Assert.Multiple definition/usages:\n'
rg -n "class Assert|static .*Multiple|Multiple\(" -g '*.cs' . | head -200Repository: dailydevops/frameshift
Length of output: 36490
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Candidate Assert classes/namespace declarations:\n'
rg -n "namespace .*Assert|class \$?Assert|struct \$?Assert|record \$?Assert|Assert\.Single|Assert\.Multiple" tests -g '*.cs' | head -200
printf '\nFiles containing Assert class keywords near declarations:\n'
rg -l "static .*Multiple|Assert\.Single|Assert\.Multiple|class .*(Assert|Assertions)|namespace .*Test" tests -g '*.cs' | head -100 | xargs -r rg -n "Assert\.Multiple\(|class .*Assert|namespace .*Test" | head -300Repository: dailydevops/frameshift
Length of output: 50378
🌐 Web query:
NUnit Assert.Multiple continue execution after failed assertion documentation
💡 Result:
In NUnit, the Assert.Multiple method is designed to allow a test to continue executing even after an assertion failure occurs, by accumulating all failures within the provided code block and reporting them together upon exit [1][2]. Key points regarding Assert.Multiple: 1. Functionality: When you wrap a series of assertions in an Assert.Multiple block, NUnit does not terminate the test immediately upon the first failure [1][3]. Instead, it records the failure, proceeds to execute the subsequent assertions in the block, and reports all accumulated failures after the block finishes [1][2]. 2. Termination behavior: The test will continue to execute until the end of the Assert.Multiple block [1][4]. If any failures were recorded during the block, the test will be marked as failed and the errors will be reported at that point [1][4]. 3. Exceptions: Assert.Multiple is intended for assertion failures (AssertionException) [5]. If an unhandled exception (other than an assertion failure) occurs within the block, the test will terminate immediately, as this typically indicates an error in the test code itself [1][4]. If an unhandled exception occurs after some assertion failures have already been recorded, NUnit will report both the recorded failures and the terminating exception [1][4]. 4. Assert.Fail: Calls to Assert.Fail within an Assert.Multiple block are handled identically to other assertion failures; the message and stack trace are recorded, and execution continues to the end of the block [1][4]. This feature is particularly useful for scenarios such as verifying multiple properties of an object or checking several UI elements where you want to see all failures at once rather than fixing them one by one [1][4].
Citations:
- 1: https://docs.nunit.org/articles/nunit/writing-tests/assertions/multiple-asserts
- 2: https://docs.nunit.org/api/NUnit.Framework.Assert.html
- 3: https://docs.dndocs.com/n/NUnit/4.3.2/api/NUnit.Framework.Assert.html
- 4: https://docs.nunit.org/articles/nunit/writing-tests/assertions/multiple-asserts.html
- 5: https://stackoverflow.com/questions/75159686/why-does-assert-multiple-remember-an-already-caught-assertion-failure
🌐 Web query:
NUnit 4.3 Assert.That IsEqualTo failure exception type
💡 Result:
In NUnit, when an assertion such as Assert.That(actual, Is.EqualTo(expected)) fails, it throws an AssertionException [1][2]. This is the standard exception type used by the NUnit framework to indicate that a test constraint has not been satisfied [1]. The AssertionException is specifically designed for this purpose and is caught by the NUnit test runner to mark the test as failed. Note that while an AssertionException is the expected result of a failed comparison, other exception types (such as System.NotSupportedException or System.MissingMethodException) might be thrown if there are configuration issues, incompatible types, or regressions during the assertion execution itself [3][4]. However, these indicate errors in the test execution or setup, rather than a standard assertion failure [3][4].
Citations:
- 1: https://learn.microsoft.com/en-us/dotnet/api/nunit.framework.assert.that
- 2: https://learn.microsoft.com/en-us/dotnet/api/nunit.framework.assert.that?view=xamarin-ios-sdk-12
- 3: Is.EqualTo with empty ValueTuple throws System.NotSupportedException "Specified Tolerance not supported for instances of type 'System.ValueTuple' and 'System.ValueTuple'" after updating to NUnit 4.2.2 nunit/nunit#4807
- 4: Breaking change in version 4.4.0-beta.1 nunit/nunit#5011
Keep dependent expressions out of grouped assertion scopes.
Assert.Multiple() continues after a failed assertion, so later expressions can throw instead of reporting the original assertion failure.
- Move collection-size assertions before the scope before indexing the collection.
- Make nullable assertions guard non-null values before accessing member properties.
📍 Affects 3 files
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutationOperatorGuardTests.cs#L194-L199(this comment)tests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TUnitTestFrameworkProbeTests.cs#L259-L263tests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TestMethodDiscoveryTests.cs#L259-L264tests/NetEvolve.FrameShift.Tests.Unit/TestSurface/TestMethodDiscoveryTests.cs#L327-L334
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutationOperatorGuardTests.cs`
around lines 194 - 199, In MutationOperatorGuardTests.cs:194-199, move the
mutations.Length assertion outside Assert.Multiple before indexing mutations,
then keep only safe grouped assertions. Apply the same dependent-expression
fixes in TUnitTestFrameworkProbeTests.cs:259-263 and
TestMethodDiscoveryTests.cs:259-264 and 327-334: perform collection-size checks
before indexing, and ensure nullable values are asserted non-null before
accessing their members; sites corrected by the shared assertion pattern require
the corresponding direct updates.
| using (Assert.Multiple()) | ||
| { | ||
| _ = await Assert.That(Targets(mutations)).IsEqualTo(Others("Ordinal")); | ||
| _ = await Assert.That(Rewrite(tree, MutationTo(mutations, "OrdinalIgnoreCase"))).IsEqualTo(expected); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Repository files around tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators:"
git ls-files | rg 'tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/.*(String).*Tests\.cs$|TUnit|Assert\.Multiple|AssertTo' || true
echo
echo "Relevant test excerpts:"
for f in tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparerMutatorTests.cs tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparisonMutatorTests.cs; do
if [ -f "$f" ]; then
echo "--- $f"
wc -l "$f"
sed -n '380,590p' "$f" | cat -n
fi
done
echo
echo "Search MutationTo and Rewriter pattern usage:"
rg -n "MutationTo\\(|Rewrite\\(|Assert\\.Multiple|Assert\\.That\\(Targets|\\.Single\\(" tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators -g '*.cs'Repository: dailydevops/frameshift
Length of output: 50378
🌐 Web query:
TUnit Assert.Multiple evaluates every assertion inside scope docs
💡 Result:
Yes, Assert.Multiple in TUnit is designed to aggregate assertions so that all failures within the scope are reported together rather than stopping at the first failure [1][2]. When you wrap assertions in a using (Assert.Multiple) block, TUnit collects the results of all awaited assertions within that scope [1]. If one or more assertions fail, TUnit accumulates these failures and reports them as a combined set when the scope is disposed [1][2]. Key behaviors to note: - Awaiting Assertions: It is critical to await each assertion (e.g., await Assert.That(...)) inside the block [1]. Because TUnit assertions are lazily evaluated, the await keyword triggers the execution; failing to await an assertion means it will not be executed or caught by the multiple assertion scope [1][3]. - Failure Reporting: By grouping related assertions, you ensure that you receive feedback on every failure in that group, which helps in debugging complex test scenarios where multiple related conditions should hold true [1]. - Scope: Assert.Multiple works for standard TUnit assertions and is compatible with other assertion extensions, such as log-based assertions, provided they follow the standard assertion execution flow [2][4]. - Limitations: Assert.Fail typically does not function as a standard assertion for accumulation purposes and may behave differently than assertion conditions; it is generally recommended to use standard assertions (like Assert.That) to ensure proper failure collection [5].
Citations:
- 1: https://tunit.dev/docs/assertions/getting-started/
- 2: https://logassertions.dev/
- 3: Problems With Asynchronous Assertions thomhurst/TUnit#3969
- 4: https://github.com/JohnVerheij/LogAssertions.TUnit/blob/main/README.md
- 5: Using Assert.Fail inside Assert.Multiple stops test after first assertion thomhurst/TUnit#2480
Make dependent mutation lookups non-throwing.
An Assert.Multiple() collects awaiting assertion failures, but it does not prevent a pre-condition MutationTo(...).Single(...) call from throwing before the rewrite assertions run. In the listed StringComparerMutatorTests and StringComparisonMutatorTests blocks, use a nullable lookup and guard Rewrite so both failures are reported together.
📍 Affects 2 files
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparerMutatorTests.cs#L423-L427(this comment)tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparerMutatorTests.cs#L436-L440tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparerMutatorTests.cs#L449-L453tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparerMutatorTests.cs#L466-L470tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparerMutatorTests.cs#L558-L562tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparisonMutatorTests.cs#L432-L436tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparisonMutatorTests.cs#L449-L453tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparisonMutatorTests.cs#L462-L466tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparisonMutatorTests.cs#L479-L483tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparisonMutatorTests.cs#L571-L575
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringComparerMutatorTests.cs`
around lines 423 - 427, Make dependent mutation lookups non-throwing in the
listed blocks using MutationTo within StringComparerMutatorTests.cs at lines
423-427, 436-440, 449-453, 466-470, and 558-562, and
StringComparisonMutatorTests.cs at lines 432-436, 449-453, 462-466, 479-483, and
571-575: retrieve the expected mutation nullable, then guard Rewrite so missing
mutations become collected assertion failures rather than precondition
exceptions while preserving the existing Assert.Multiple assertions.
| using (Assert.Multiple()) | ||
| { | ||
| _ = await Assert.That(mutations).Count().IsEqualTo(1); | ||
| _ = await Assert.That(mutations[0].Kind).IsEqualTo(MutationKind.StringLiteral); | ||
| _ = await Assert.That(mutations[0].OperatorId).IsEqualTo("string-literal.to-empty"); | ||
| _ = await Assert.That(mutations[0].DisplayName).IsEqualTo("\"...\" => \"\""); | ||
| _ = await Assert.That(Rewrite(tree, mutations[0])).IsEqualTo(EmptySource); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Locate test files and assert multiple usages =="
git ls-files | rg '(^|/)StringLiteralMutatorTests\.cs$|(^|/)UnaryOperatorMutatorTests\.cs$|(^|/)MutantCompilerTests\.cs$|(^|/)RegexPatternTokenizerAdversarialTests\.cs$|(^|/)RegexPatternTokenizerTests\.cs$' || true
echo
echo "== Relevant Assert.Multiple usages with context =="
for f in \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/UnaryOperatorMutatorTests.cs \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutantCompilerTests.cs \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerAdversarialTests.cs \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerTests.cs
do
echo "--- $f lines 90-340 ---"
sed -n '90,340p' "$f" | nl -ba -v90
done
echo
echo "== Project/package refs for TUnit =="
rg -n 'TUnit|Using TUnit|<PackageReference|<ProjectReference' -S . --glob '*.csproj' --glob '*.props' --glob '*.targets' || trueRepository: dailydevops/frameshift
Length of output: 862
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Read relevant test ranges =="
for f in \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/UnaryOperatorMutatorTests.cs \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutantCompilerTests.cs \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerAdversarialTests.cs \
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerTests.cs
do
echo "--- $f ---"
awk 'NR>=90 && NR<=340 { printf "%6d\t%s\n", NR, $0 }' "$f"
done
echo
echo "== Search Assert.Multiple usages around collection assertions =="
rg -n -C4 'Assert\.Multiple|\.Count\(\)|\.Length|First\(\)|\[0\]|\[1\]|Substring' tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/UnaryOperatorMutatorTests.cs tests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutantCompilerTests.cs tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerAdversarialTests.cs tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerTests.cs || true
echo
echo "== Project/package refs for TUnit =="
rg -n 'TUnit|<PackageReference|<ProjectReference' -S . --glob '*.csproj' --glob '*.props' --glob '*.targets' || true
echo
echo "== Repository files around TUnit package =="
find . -maxdepth 4 \( -name '*.csproj' -o -name '*.props' -o -name '*.targets' -o -name '*.config' \) -print | sort | xargs -r grep -n 'TUnit' || trueRepository: dailydevops/frameshift
Length of output: 50379
🌐 Web query:
TUnit Assert.Multiple evaluated every assertion fails subsequent assertions still execute documentation
💡 Result:
In TUnit, using Assert.Multiple allows you to group multiple assertions so that they all execute, regardless of whether previous assertions within the block fail [1]. When an assertion inside an Assert.Multiple block fails, TUnit collects the failure rather than throwing an exception immediately, allowing subsequent assertions in that block to proceed [2][1]. Once the block is disposed at the end of the using statement, TUnit aggregates all accumulated failures and reports them together [2][1]. To ensure this behavior works correctly, you must await your assertions (e.g., await Assert.That(...)) [3][1]. Unawaited assertions inside an Assert.Multiple block may not be properly registered or executed, which can lead to silent passes or unexpected behavior [3][1]. Note that historically, certain methods like Assert.Fail did not behave this way when called inside an Assert.Multiple block, potentially causing immediate termination [4]. However, TUnit is designed to support the accumulation of failures for standard assertions within these scopes [2][1]. If you encounter issues where execution seems to stop prematurely, ensure you are using standard fluent assertions (Assert.That) correctly and awaiting them [1].
Citations:
- 1: https://tunit.dev/docs/assertions/getting-started/
- 2: fix: IsNotNull assertion thomhurst/TUnit#3634
- 3: Problems With Asynchronous Assertions thomhurst/TUnit#3969
- 4: Using Assert.Fail inside Assert.Multiple stops test after first assertion thomhurst/TUnit#2480
Make collection-dependent assertions run only after size checks.
Assert.Multiple() accumulates failures, so an IsEqualTo(1) / IsEmpty() / index < pattern.Length failure can still be followed by indexing, First(), or Substring(). Move the dependent access behind the length/count guard, or replace First() with a non-indexing predicate, so diagnostic failures are reported instead of masked by throw exceptions.
📍 Affects 5 files
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs#L103-L110(this comment)tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs#L118-L127tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs#L134-L139tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs#L152-L159tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs#L173-L180tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs#L206-L213tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs#L221-L228tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs#L327-L333tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/UnaryOperatorMutatorTests.cs#L402-L408tests/NetEvolve.FrameShift.Tests.Unit/Mutations/MutantCompilerTests.cs#L285-L291tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerAdversarialTests.cs#L463-L469tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerAdversarialTests.cs#L918-L925tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternTokenizerTests.cs#L546-L553
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tests/NetEvolve.FrameShift.Tests.Unit/Mutations/Operators/StringLiteralMutatorTests.cs`
around lines 103 - 110, The collection-dependent assertions must not access
elements or derive substrings while their count, emptiness, or index
preconditions are only asserted inside Assert.Multiple(); guard those accesses
or use non-indexing predicates so assertion failures remain diagnostic. Apply
this to StringLiteralMutatorTests.cs ranges 103-110, 118-127, 134-139, 152-159,
173-180, 206-213, 221-228, and 327-333; UnaryOperatorMutatorTests.cs 402-408;
MutantCompilerTests.cs 285-291; RegexPatternTokenizerAdversarialTests.cs 463-469
and 918-925; and RegexPatternTokenizerTests.cs 546-553.
| using (Assert.Multiple()) | ||
| { | ||
| _ = await Assert.That(site.Origin).IsEqualTo(RegexPatternOrigin.RegexConstructor); | ||
| _ = await Assert.That(site.Pattern).IsEqualTo("a+"); | ||
| _ = await Assert.That(site.AreOptionsKnown).IsTrue(); | ||
| _ = await Assert.That(site.Options!.Value).IsEqualTo(RegexOptions.None); | ||
| _ = await Assert.That(site.OptionsExpression).IsNull(); | ||
| _ = await Assert.That(site.AttributeArgument).IsNull(); | ||
| _ = await Assert.That(site.PatternLiteral.Token.ValueText).IsEqualTo("a+"); | ||
| } | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate file =="
fd -a 'RegexPatternLocatorTests.cs' . || true
echo "== target lines =="
if [ -f tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs ]; then
sed -n '380,1135p' tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs | nl -ba -v380
fi
echo "== assert multiple / nullable docs references in repo =="
rg -n "Assert\.Multiple|multiple|Assertion.*diagnostic|TUnit|Assert\.That.*Value|OptionsExpression?.*ToString|AttributeArgument" -S .Repository: dailydevops/frameshift
Length of output: 351
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate assert multiple references across repo =="
rg -n "Assert\.Multiple|Multiple\(\)|multiple assertion|all assertions run|get all assertion" -S --glob '!bin/**' --glob '!obj/**' . || true
echo "== inspect Assert extension / helper definitions =="
rg -n "namespace .*|class.*Assert|record.*Assert|static .*Assert|Multiple|IsEqualTo|IsNull|IsNotNull|HasValue" -S --glob '!bin/**' --glob '!obj/**' . || trueRepository: dailydevops/frameshift
Length of output: 50379
🌐 Web query:
TUnit Assert.Multiple all assertions in scope run and then failures reported
💡 Result:
In TUnit, Assert.Multiple acts as an assertion scope that allows multiple assertions to be executed even if preceding ones within the same block have failed [1][2]. When assertions are placed within a using (Assert.Multiple) block, TUnit tracks and aggregates any assertion failures that occur [3][2], ensuring that all code in the scope is executed rather than stopping at the first failure [1]. Key behavioral details: - Aggregation: Upon exiting the scope (disposal of the scope object), TUnit reports all accumulated assertion failures together, providing a complete view of all issues encountered [3][2]. - Execution Flow: Unlike standard assertion execution, which throws an exception immediately upon failure, the Assert.Multiple scope suppresses immediate termination to allow subsequent assertions to run [3]. - Asynchronous Usage: Since TUnit relies on the await keyword to trigger assertion execution [4], you must ensure that assertions within the scope are properly awaited (e.g., using await Assert.That(...)...) [1][2]. If not properly awaited, the assertion might not be registered correctly within the multiple scope [3]. - Limitations: Assert.Fail historically caused early termination in some versions, but this behavior has been a focus of updates to ensure consistency within multiple scopes [1]. Always ensure your assertion library version is up to date to benefit from the latest fixes regarding assertion scoping and failure handling [3]. This mechanism is particularly useful for verifying multiple properties of a system under test (e.g., return values combined with side effects like logs) [2][5].
Citations:
- 1: Using Assert.Fail inside Assert.Multiple stops test after first assertion thomhurst/TUnit#2480
- 2: https://github.com/JohnVerheij/LogAssertions.TUnit
- 3: fix: IsNotNull assertion thomhurst/TUnit#3634
- 4: Problems With Asynchronous Assertions thomhurst/TUnit#3969
- 5: https://logassertions.dev/
🌐 Web query:
NullReferenceException Assert.Multiple TUnit earlier assertion failure runtime exception
💡 Result:
In the TUnit testing framework, a NullReferenceException occurring within an Assert.Multiple block is typically not an intended behavior of the framework itself but rather a side effect of how certain assertions or user code interact with the assertion pipeline [1][2][3]. Key insights into why this may occur include: 1. Bypassing the Assertion Pipeline: In earlier versions of TUnit, some assertions (such as IsNotNull) contained custom logic that bypassed the standard assertion execution flow (specifically by not calling AssertAsync correctly) [1]. This meant that if the assertion failed, it might not throw a controlled AssertionException, potentially leading to subsequent NullReferenceExceptions when the code proceeded with an invalid state [1]. 2. Lack of Null Guards in Generic Assertions: Some assertions, particularly certain comparison-based assertions, have historically lacked explicit null checks [2]. When these generic assertions receive a null value, the underlying C# operations (such as CompareTo) can throw a NullReferenceException before the assertion framework can intercept it and generate a descriptive AssertionException [2]. 3. Implicit Conversion Issues: There have been identified issues where user-defined implicit conversions (e.g., implicit to string) trigger a NullReferenceException at the call site before the TUnit assertion logic is even reached [4]. This happens when the framework's overload resolution interacts with these conversions on a null receiver [4]. 4. Improper Assertion Chaining: Because TUnit assertions are built lazily (they do not execute until awaited), building a complex chain inside Assert.Multiple without proper awaiting or handling of pending pre-work can lead to unexpected execution orders or unvalidated state [3][5]. Ensuring all assertions are properly awaited is critical to maintaining the integrity of the assertion scope [6][5]. Recommended Troubleshooting: - Ensure you are using the latest version of TUnit, as many of these issues have been addressed in recent patches [1][4][3][7]. - Verify that all assertions are properly awaited within the Assert.Multiple scope [6]. - If the exception persists, check if any custom types or implicit conversions are involved in the failing assertion, as these are common sources of pre-assertion NullReferenceExceptions [4]. Top results: [1][2][4][3][5]
Citations:
- 1: fix: IsNotNull assertion thomhurst/TUnit#3634
- 2: refactor: convert 15 manual assertions to [GenerateAssertion] thomhurst/TUnit#5029
- 3: fix(assertions): ensure previous assertions are validated before type conversion thomhurst/TUnit#3427
- 4: fix(assertions): prevent implicit-to-string op from NREing on null (#5692) thomhurst/TUnit#5696
- 5: Problems With Asynchronous Assertions thomhurst/TUnit#3969
- 6: https://tunit.dev/docs/getting-started/writing-your-first-test/
- 7: [Bug]: 5 small papercuts on 1.36.0 found during MSTest→TUnit migration (assertion diagnostics, array equality, generic inference, analyzer ergonomics) thomhurst/TUnit#5613
🌐 Web query:
site.Options.HasValue site.OptionsExpression OptionsLiteral RegexOptions
💡 Result:
The terms you listed appear to be from different, unrelated contexts within the.NET ecosystem, rather than a single unified API or class. Here is the breakdown of these components: 1. RegexOptions: This is a well-known enumeration in the System.Text.RegularExpressions namespace [1][2]. It provides bitwise flags (such as IgnoreCase, Compiled, or Singleline) used to modify the behavior of regular expression matching [1][3][4]. 2. site.Options.HasValue: This pattern is not part of a standard.NET library. However, the HasValue property is a common idiom in.NET for checking if an optional or nullable type contains a value. For example, it is used in Nullable, F# option types [5], or custom Option wrapper classes used in various frameworks [6][7]. The prefix site. suggests this might be specific to a custom application or a third-party framework (such as a CMS or site-configuration library). 3. OptionsExpression and OptionsLiteral: These are not standard.NET class names. They likely represent custom implementations within a specific library or application that handles configuration, parsing, or command-line arguments [8]. If you encountered these together, they are likely from a custom codebase, a specific framework (like a custom site-management tool), or perhaps a generated configuration object where OptionsExpression might represent a programmatic way to define options and OptionsLiteral might represent a hard-coded value. To identify the exact source, please check your project's documentation or use a tool like "Go to Definition" in your IDE to locate the assembly or namespace defining these specific members.
Citations:
- 1: https://learn.microsoft.com/en-us/dotnet/api/system.text.regularexpressions.regexoptions?view=net-8.0
- 2: https://source.dot.net/System.Text.RegularExpressions/System/Text/RegularExpressions/RegexOptions.cs.html
- 3: https://learn.microsoft.com/en-us/dotnet/api/system.text.regularexpressions.regexoptions?view=net-9.0
- 4: https://github.com/dotnet/dotnet/blob/b0f34d51fccc69fd334253924abd8d6853fad7aa/src/runtime/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexOptions.cs
- 5: https://learn.microsoft.com/en-us/dotnet/fsharp/language-reference/options
- 6: https://source.dot.net/dotnet-svcutil-lib/Shared/Options/OptionBase.cs.html
- 7: https://academy.creatio.com/api/netcoreapi/8.0.0/api/Terrasoft.Common.Option-1.html
- 8: https://learn.microsoft.com/en-us/dotnet/api/system.commandline.option-1?view=net-10.0-pp
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target file header and relevant type definitions =="
sed -n '1,80p' tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs
echo
echo "== locate Regexp*Locator definitions =="
fd -a -i 'regex.*locator|regexp.*locator|locator' . | sed 's#^\./##' | head -50
echo
echo "== type outlines =="
fd -a -i 'regex.*locator|regexp.*locator|locator' . -x sh -c 'echo "--- $1"; ast-grep outline "$1" --view compressed 2>/dev/null | head -200 || true' sh {}Repository: dailydevops/frameshift
Length of output: 4699
🌐 Web query:
TUnit Assertions 6.9 NullReferenceException scope disposal assertion failure execution order Assert.AssertionExecutionScope 3634
💡 Result:
The reference to "TUnit Assertions 6.9" and "AssertionExecutionScope 3634" relates to a specific fix in TUnit's assertion pipeline, specifically addressed in pull request #3634 [1]. In earlier versions of TUnit, certain assertions—most notably IsNotNull()—implemented custom GetAwaiter() logic that bypassed the standard assertion execution pipeline [1]. This bypass caused several issues, particularly when using Assert.Multiple() (the assertion scope) [1]: 1. Improper Scope Behavior: Because the assertion bypassed the standard AssertAsync() flow, it would fail to accumulate failures when inside an Assert.Multiple() scope and fail to throw immediately when outside of one [1]. 2. NullReferenceException Risk: By bypassing the validation pipeline, these assertions could return null values to subsequent code, leading to unexpected NullReferenceExceptions elsewhere in the execution flow [1]. The fix in PR #3634 updated the IsNotNull implementation to call AssertAsync() instead of directly accessing the Context [1]. This change ensures that: - Validation logic (CheckAsync()) is correctly executed [1]. - The assertion adheres to Assert.Multiple() scoping rules, correctly accumulating failures or throwing as expected [1]. - Nullability is handled within the formal pipeline, mitigating the risk of downstream NullReferenceExceptions [1]. This architectural improvement reflects TUnit’s broader design, where assertions are lazily executed (triggered by await) to allow for fluent chain building [2], and where the AssertionScope is used to aggregate failures across multiple checks [1][3]. Other related issues in TUnit have involved preventing NullReferenceExceptions during implicit-to-string conversions [4] and fixing race conditions in factory disposal that caused similar NREs [5], but the specific reference to assertion scope disposal failure and order is corrected by ensuring all assertions consistently route through the centralized AssertAsync() execution flow [1][6].
Citations:
- 1: fix: IsNotNull assertion thomhurst/TUnit#3634
- 2: Problems With Asynchronous Assertions thomhurst/TUnit#3969
- 3: Simplify Assertion API thomhurst/TUnit#3324
- 4: fix(assertions): prevent implicit-to-string op from NREing on null (#5692) thomhurst/TUnit#5696
- 5: fix(aspnetcore): serialize WithWebHostBuilder to stop _derivedFactories race (flaky disposal NRE) thomhurst/TUnit#6251
- 6: perf: improve
Assertion.ExecuteCorestate machine generation thomhurst/TUnit#4328
Use assertion shapes that cannot fail before reaching intended diagnostics.
TUnit continues executing assertions inside Assert.Multiple; when a failed assertion or expression evaluation throws before the intended check is reached, later nullable-safe assertions cannot replace that failure. Choose assertion forms that do not make earlier assertions or .Value, !, or Parent calls a precondition for later assertions.
📍 Affects 1 file
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L421-L431(this comment)tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L438-L445tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L461-L468tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L475-L481tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L494-L503tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L510-L514tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L526-L532tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L547-L553tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L565-L571tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L591-L596tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L618-L625tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L659-L666tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L687-L692tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L702-L708tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L838-L843tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L951-L960tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L972-L979tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L989-L995tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L1005-L1009tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L1042-L1048tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L1086-L1094tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs#L1107-L1113
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs`
around lines 421 - 431, Update the Assert.Multiple blocks in
RegexPatternLocatorTests.cs at
tests/NetEvolve.FrameShift.Tests.Unit/Mutations/RegularExpressions/RegexPatternLocatorTests.cs:421-431,
438-445, 461-468, 475-481, 494-503, 510-514, 526-532, 547-553, 565-571, 591-596,
618-625, 659-666, 687-692, 702-708, 838-843, 951-960, 972-979, 989-995,
1005-1009, 1042-1048, 1086-1094, and 1107-1113 to use assertion shapes that
remain independently evaluable when earlier checks fail; avoid
nullable-forgiving access, Nullable<T>.Value, or Parent/property dereferences
that can throw before later assertions report their intended diagnostics.
Wrap consecutive TUnit assertions in using (Assert.Multiple()) so a failing assertion no longer hides the remaining checks in the same test, making failures easier to diagnose in one run.
1 participant
Summary
using (Assert.Multiple()) { }across the unit and integration test suites, so a failing assertion no longer masks the remaining checks in the same test run.Test plan
dotnet buildonNetEvolve.FrameShift.Tests.Unitdotnet buildonNetEvolve.FrameShift.Tests.IntegrationSummary by CodeRabbit