Skip to content

Skip.Test custom reason lost #3441

@nil4

Description

@nil4

Related to #3406 and #3415, a couple issues became apparent with TUnit 0.73.14:

  • when using Skip.Test(customReason) at runtime, that custom reason is lost and never reported, either by TUnit's built-in reporter or by attribute-based event receivers
  • such tests are also reported as finished twice.

Steps to reproduce

> dotnet --version
10.0.100-rc.2.25502.107

> mkdir repro && cd repro
> dotnet new console
> dotnet add package TUnit@0.73.14

Replace Program.cs with:

using TUnit.Core.Interfaces;

[assembly: ReproReporter]

class C {
    [Test] public void SkippedAtRuntime() => Skip.Test("CustomReason");
}

[AttributeUsage(AttributeTargets.Assembly)]
class ReproReporterAttribute : Attribute, ITestEndEventReceiver, ITestSkippedEventReceiver
{
    public ValueTask OnTestSkipped(TestContext ctx)
        => Log($"testSkipped {ctx.GetDisplayName()} (Reason={ctx.SkipReason})");

    public ValueTask OnTestEnd(TestContext ctx)
        => Log($"testDone {ctx.GetDisplayName()} (Reason={ctx.SkipReason})");

    static ValueTask Log(string message) {
        GlobalContext.Current.OriginalConsoleOut.WriteLine($"##report[{message}]");
        return ValueTask.CompletedTask;
    }
}

Run and observe the actual output:

##report[testDone SkippedAtRuntime (Reason=)]
##report[testSkipped SkippedAtRuntime (Reason=)]
##report[testDone SkippedAtRuntime (Reason=)]
skipped SkippedAtRuntime (0ms)
  Skipped

Test run summary: Zero tests ran - bin/Debug/net10.0/repro.dll (net10.0|arm64)
  total: 1
  failed: 0
  succeeded: 0
  skipped: 1

Issues observed

  1. The skip reason passed to Skip.Test("CustomReason") is lost and not reported.

  2. testDone SkippedAtRuntime is reported twice. This makes it difficult to accurately report when tests start and finish to CI systems.

Expected output

-##report[testDone SkippedAtRuntime (Reason=)]
-##report[testSkipped SkippedAtRuntime (Reason=)]
+##report[testSkipped SkippedAtRuntime (Reason=CustomReason)]
-##report[testDone SkippedAtRuntime (Reason=)]
+##report[testDone SkippedAtRuntime (Reason=CustomReason)]
skipped SkippedAtRuntime (0ms)
-  Skipped
+  CustomReason

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions