-
-
Notifications
You must be signed in to change notification settings - Fork 98
Description
I've spend many hours in frustration trying to understand what is going on and why I can't filter out performance tests on CI.
Eventually I came up with a simple small test project like this:
public class TestClass1
{
[Test]
[Category("Performance")]
public async Task TestClass1TestMethod1()
{
Console.WriteLine("TestClass1TestMethod1");
}
[Test]
[Property("CI", "false")]
public async Task TestClass1TestMethod2()
{
Console.WriteLine("TestClass1TestMethod2");
}
}
public class TestClass2
{
[Test]
[Category("Performance")]
[Property("CI", "false")]
public async Task TestClass2TestMethod1()
{
Console.WriteLine("TestClass2TestMethod1");
}
[Test]
[Explicit] // !!!!
public async Task TestClass2TestMethod2()
{
Console.WriteLine("TestClass2TestMethod2");
}
}
There is very big difference weather any test marked as [Explicit] exists or not.
If there is no explicit, those filters works as expected:
/*/*/*/*[Category=Performance]
/*/*/*/*[Category!=Performance]
But if I add [Explicit] attribute:
/*/*/*/*[Category=Performance]
/*/*/*/*[Category!=Performance] <-- does not. Now it runs all non-explicit tests, even with Performance category
So, Explicit works alone, Inclusive category filters works alone, but negative category filters with Explicit behaves unexpectedly... attempt to add Explicit test breaks CI due to wrong set of tests being executed.