Enhance Dynamic Tests

[PandrPi]

Hello @thomhurst and all who may find it interesting,

I have specific requirements that I want to resolve using TUnit (which looks fantastic, by the way, and very promising).

I need a solution for testing an async tree-like structure, which may vary from case to case. The following example will be mostly made up because of company policy, but I will try to make it as close to the real case as possible.

I have a bunch of devices from which I can request the menu structure. Under the hood, all the data is requested via REST API, so I need to use async logic. The menu structure may look the following:

Name: MainMenu
Data:
  - Data1
  - Data2
  - Data3
Children:
  - Name: SubmenuA
    Data:
      - Data4
      - Data5
    Children: []
  - Name: SubmenuB
    Data:
      - Data6
    Children:
      - Name: SubmenuC
        Data:
          - Data7
          - Data8
          - Data9
      - Name: SubmenuD
        Data:
          - Data10
          - Data11

On the C# side, the nodes are defined in the following way:

public interface INode
{
    Task<string> GetNameAsync(CancellationToken token);
    IAsyncEnumerable<object> GetDatasAsync(CancellationToken token);
    IAsyncEnumerable<INode> GetChildrenAsync(CancellationToken token);
}

The implementation of all this stuff does not actually matter in this post.

I want to automate the testing of these menu structures independently of a device. The problem is that the menu structure itself is usually very different from device to device. I cannot implement a bunch of tests that verify all the possible menu structures, their submenus, and the data of those submenus. It just won't work most of the time.

Instead, I want to connect to a device before running the tests, figure out what menus and submenus we have, and then create tests dynamically in the runtime for verifying the data of the menus/submenus.

https://tunit.dev/docs/experimental/dynamic-tests looks very promising and it's basically what I need.
The only problem I find in the approach of DynamicTest is that it's not really async (at least it can't be as of v0.19.32).

DynamicTestBuilder attribute expects only a sync test method, while DunamicTest.TestMethod is an Expression<Action<TClass>>. So, in both solutions, there can't be any async test methods, which is not good for my case.

How would you define a solution for asynchronously creating dynamic tests with asynchronous test methods without blocking anything?
Is there any other potential solution?

[thomhurst]

Right now, no. But the plan is it'll allow sync and Async. I was trying out a way with an expression to just point to the method, to get a method info, and not actually care if it's Async or not, but it didn't quite work how I envisioned. I might just have to settle for two properties. Sync and Async.

[PandrPi]

Well then. I hope you will find time to look at this. If there is anything I can help with or contribute, please let me know. Thank you!

[thomhurst]

@PandrPi I've taken a look this evening. Because it's only an expression it should still work with async methods! It just uses the expression to get a pointer to the method. If it's a task, the engine should be able to handle it.

[PandrPi]

I verified this with a simple async test containing only await Task.Delay(10_000).ConfigureAwait(false) and the test actually took ~10 seconds to execute, meaning that you are absolutely right about this statement!

The only moment I would suggest is to fix the syntax warning saying, because this call is not awaited... bla bla bla.. Maybe you can introduce the TestMethodAsync property to the DynamicTest<T> class, as you mentioned before. Or, perhaps even easier, would be to suppress this warning by introducing a new Roslyn Analyzer rule.

What way do you prefer more?

[thomhurst]

I did write an analyzer to try and suppress this: https://github.com/thomhurst/TUnit/blob/main/TUnit.Analyzers/DynamicTestAwaitExpressionSuppressor.cs

I guess it's not kicking in?

[PandrPi]

For some reason, it does not suppress the warning after I updated the nuget to v0.19.52. I have to manually disable it with #pragma