Using shared data source (1.0.78)

[JaneMalico]

I want each test to have its own instance of WebApplicationFactory (so I can mock different behaviors per test) while sharing instances of the test containers, since they are resource-heavy.

To inject the test container instances, I use ClassDataSourceAttribute with SharedType.PerTestSession. My understanding is that the containers should be initialized only once per test session (i.e. be shared among test method defined in the same class).

However, it seems that each test starts its own instance of the test container instead. Am I missing something?

1)The test container classes look as follows

public sealed class TestRabbitContainer : IAsyncInitializer, IAsyncDisposable
{
    private readonly RabbitMqContainer _instance = new RabbitMqBuilder().Build();

    public string ConnectionString => _instance.GetConnectionString();
    public async Task InitializeAsync() => await _instance.StartAsync();
    public async ValueTask DisposeAsync() => await _instance.DisposeAsync();
}

2. The test containers are injected into the WebApplicationFactory ( SharedType.PerTestSession)

public class WebApplicationFactory : WebApplicationFactory<Program>, IAsyncInitializer
{
    [ClassDataSource<TestRabbitContainer>(Shared = SharedType.PerTestSession)]
    public required TestRabbitContainer TestRabbitContainer { get; init; }

    [ClassDataSource<TestSqlContainer>(Shared = SharedType.PerTestSession)]
    public required TestSqlContainer TestSqlContainer{ get; init; }

    // Ensures the app is started without creating client
    public Task InitializeAsync()
    {
        _ = Server;
        return Task.CompletedTask;
    }
...

3. The WebApplication is injected into the test class

public class MyTests
{

    [ClassDataSource<WebApplicationFactory>]
    public required WebApplicationFactory WebApplicationFactory { get; init; }

    private const int DefaultTestTimeoutMilliseconds = 17_000;

    [Test]
    [Timeout(DefaultTestTimeoutMilliseconds)]
    public async Task Test1(CancellationToken cancellationToken)
    {
        await Assert.That(Random.Shared.Next()%2).IsEqualTo(0);
    }

    [Test]
    [Timeout(DefaultTestTimeoutMilliseconds)]
    public async Task Test2(CancellationToken cancellationToken)
    {
        await Assert.That(Random.Shared.Next() % 2).IsEqualTo(0);
    }
}

I have two tests that each create their own containers, even when run in the same session.

When I downgrade TUnit to version 0.67.19, only one instance of each container is created - which appears to be the expected behavior

[thomhurst]

@JaneMalico Try v1.1.0

[JaneMalico]

Unfortunately, it seems to be non-deterministic. It works perfectly fine with your test examples, but as soon as InitializeAsync and the resource creation become time-consuming (for example, initializing a container field or running a lengthy initialization method), the constructor is sometimes called twice. This leads to an error in my case, since the actual code uses the same port binding.

I also run tests in parallel, and in my setup, the WebApplicationFactory is not initialized with Shared.PerTestSession, although the containers are.

Another thing I noticed is that if the InitializeAsync method takes a long time (it’s easy to reproduce by adding await Task.Delay(...)), the console output from the test container’s constructor is not shown in the test output. That’s why, to confirm that the constructor was being called multiple times, I had to write to a file from within the constructor in my real scenario, because no information from Console.WriteLine was available.) It behaves the same way when I run the tests both from Visual Studio and from the console.

[thomhurst]

Test data is constructed before tests so doesn't have a test context available yet to redirect output to when you're in the constructor. I'll have a think and see if there's a way around that.

As for the constructor being called twice, that's an odd one. Would you be able to create me a minimal repro that I could clone?

[JaneMalico]

Oh, okay, I assumed the output should work if it’s implemented like that in the tests XD.
While debugging, I noticed that TestContext.Current is null in the constructor, but I figured maybe I misunderstood something, or that some global context is supposed to be used.

Unfortunately, the issue with the constructor running twice is pretty inconsistent. I can’t reproduce it myself now (been trying for half an hour), even though it was happening quite regularly before. Maybe I’m doing something wrong.

On a completely different note, I noticed that the Timeout and Arguments attributes seem to interfere with each other. Timeout works as expected when used without Arguments (cancellation is triggered on the token passed to the test), but when combined with Arguments, cancellation never happens. Maybe I’m misinterpreting how it’s supposed to work, but I was surprised to see my tests “hang” and never time out, even though I passed a cancellation token to my app.

namespace TunitTestProject
{
    internal sealed class MyTestClass
    {
        [Test]
        [Arguments(1)]
        [Arguments(2)]
        [Timeout(2000)]
        public async Task TimeoutWithArguments(int argument, CancellationToken cancellationToken)
        {
            await Assert.ThrowsAsync<TaskCanceledException>(() => Task.Delay(TimeSpan.FromSeconds(4), cancellationToken));
        }

        [Test]
        [Timeout(2000)]
        public async Task TimeoutWithoutArguments(CancellationToken cancellationToken)
        {
            await Assert.ThrowsAsync<TaskCanceledException>(() => Task.Delay(TimeSpan.FromSeconds(4), cancellationToken));
        }
    }
}

[thomhurst]

Ooh interesting. Another one for the list. You make a great QA haha