LightBDD Integration

[dpotter895]

A great tool for acceptance and regression testing is LightBDD. It would be amazing if we could use these 2 at the same time.

There currently is an open issue in LightBDD for this integration:
LightBDD/LightBDD#373

I don't know enough about TUnit to see if his checklist is simple with TUnit or not. Would you be able to look at this to see if this would be straightforward to integrate based on the above LightBDD Issue comments?

[dpotter895]

I don't know if this helps but I used AI too see if it could figure this out.

LightBDD Integration with TUnit: Implementation Approach

Based on my investigation of both the GitHub issue #373 and TUnit's codebase, here's how TUnit could fulfill Suremaker's requirements for LightBDD integration.

Core Integration Requirements

Suremaker (LightBDD maintainer) outlined three key requirements for TUnit integration:

1. Test discovery mechanism
2. Way to execute tests
3. Mechanism to capture test execution status

Let's explore how TUnit's architecture can fulfill these requirements with specific implementation details:

1. Test Discovery Integration

TUnit already has a robust test discovery mechanism through its TestDiscoverer class, which would need enhancements to support LightBDD:

namespace TUnit.Extensions.LightBDD
{
    public class LightBDDTestDiscoverer : ITestDiscoverer
    {
        private readonly ITestMethodFilter _testMethodFilter;
        
        public LightBDDTestDiscoverer(ITestMethodFilter testMethodFilter)
        {
            _testMethodFilter = testMethodFilter;
        }
        
        public IEnumerable<TestMethod> DiscoverTests(Assembly assembly)
        {
            // Find classes with FeatureDescription attribute (LightBDD feature classes)
            var featureClasses = assembly.GetTypes()
                .Where(type => type.GetCustomAttribute<FeatureDescriptionAttribute>() != null);
                
            foreach (var featureClass in featureClasses)
            {
                // Find methods with Scenario attribute (LightBDD scenarios)
                var scenarioMethods = featureClass.GetMethods()
                    .Where(method => method.GetCustomAttribute<ScenarioAttribute>() != null)
                    .Where(method => _testMethodFilter.ShouldInclude(method));
                    
                foreach (var scenarioMethod in scenarioMethods)
                {
                    // Create a specialized LightBDD TestMethod
                    yield return new LightBDDTestMethod(featureClass, scenarioMethod);
                }
            }
        }
    }
    
    // Specialized TestMethod for LightBDD scenarios
    public class LightBDDTestMethod : TestMethod
    {
        public string FeatureDescription { get; }
        public string ScenarioTitle { get; }
        public IReadOnlyList<string> StepNames { get; }
        
        public LightBDDTestMethod(Type classType, MethodInfo methodInfo) : base(classType, methodInfo)
        {
            // Extract FeatureDescription from class
            var featureAttr = classType.GetCustomAttribute<FeatureDescriptionAttribute>();
            FeatureDescription = featureAttr?.Description;
            
            // Extract Scenario title from method
            var scenarioAttr = methodInfo.GetCustomAttribute<ScenarioAttribute>();
            ScenarioTitle = scenarioAttr?.Title ?? methodInfo.Name.ToSentenceCase();
            
            // Analyze method body to extract step names (more complex in real implementation)
            StepNames = ExtractStepNames(methodInfo);
        }
        
        private List<string> ExtractStepNames(MethodInfo methodInfo)
        {
            // This would use more sophisticated techniques like IL analysis
            // to extract the step method references from RunScenarioAsync calls
            // Simplified version shown here
            return new List<string>();
        }
    }
}

2. Test Execution Integration

TUnit's TestRunner would need to be extended to support LightBDD's execution model:

namespace TUnit.Extensions.LightBDD
{
    public class LightBDDTestRunner : ITestRunner
    {
        private readonly ITestResultAggregator _resultAggregator;
        
        public LightBDDTestRunner(ITestResultAggregator resultAggregator)
        {
            _resultAggregator = resultAggregator;
        }
        
        public async Task<TestSuiteResult> RunTestsAsync(IEnumerable<TestMethod> testMethods)
        {
            var results = new List<TestResult>();
            
            foreach (var testMethod in testMethods)
            {
                if (testMethod is LightBDDTestMethod lightBDDTest)
                {
                    var result = await ExecuteLightBDDScenarioAsync(lightBDDTest);
                    results.Add(result);
                }
                else
                {
                    var result = await ExecuteStandardTestAsync(testMethod);
                    results.Add(result);
                }
                
                _resultAggregator.AddResult(results.Last());
            }
            
            return new TestSuiteResult(results);
        }
        
        private async Task<TestResult> ExecuteLightBDDScenarioAsync(LightBDDTestMethod lightBDDTest)
        {
            // Create instance of feature class
            var featureInstance = Activator.CreateInstance(lightBDDTest.ClassType);
            
            if (featureInstance is IFeatureFixture featureFixture)
            {
                // Initialize LightBDD feature fixture
                var scenarioResult = new LightBDDScenarioResult(lightBDDTest);
                
                try
                {
                    // Access the Runner property to get LightBDD's scenario runner
                    var scenarioRunner = featureFixture.Runner;
                    
                    // Execute the scenario method which will call RunScenarioAsync
                    var stopwatch = Stopwatch.StartNew();
                    var task = lightBDDTest.MethodInfo.Invoke(featureInstance, null) as Task;
                    if (task != null)
                    {
                        await task;
                    }
                    stopwatch.Stop();
                    
                    // Retrieve step results from the runner's context
                    scenarioResult.StepResults = ExtractStepResultsFromContext(scenarioRunner);
                    scenarioResult.Duration = stopwatch.Elapsed;
                    
                    return scenarioResult;
                }
                catch (Exception ex)
                {
                    scenarioResult.Status = TestStatus.Failed;
                    scenarioResult.Exception = ex;
                    return scenarioResult;
                }
            }
            
            return new TestResult(lightBDDTest, TestStatus.Skipped, 
                new InvalidOperationException("Feature class does not implement IFeatureFixture"));
        }
        
        private List<StepResult> ExtractStepResultsFromContext(object scenarioRunner)
        {
            // This would access the LightBDD context to extract step results
            // Implementation details depend on LightBDD's public API
            return new List<StepResult>();
        }
    }
}

3. Test Result Capture Integration

TUnit's TestResult and TestResultAggregator would need extensions to support LightBDD's result model:

namespace TUnit.Extensions.LightBDD
{
    public class LightBDDScenarioResult : TestResult
    {
        public List<StepResult> StepResults { get; set; } = new List<StepResult>();
        
        public LightBDDScenarioResult(LightBDDTestMethod testMethod) 
            : base(testMethod, TestStatus.Passed)
        {
            // Add LightBDD-specific metadata
            Metadata["FeatureDescription"] = ((LightBDDTestMethod)TestMethod).FeatureDescription;
            Metadata["ScenarioTitle"] = ((LightBDDTestMethod)TestMethod).ScenarioTitle;
        }
    }
    
    public class StepResult
    {
        public string StepName { get; }
        public TestStatus Status { get; set; }
        public Exception Exception { get; set; }
        public TimeSpan Duration { get; set; }
        
        public StepResult(string stepName)
        {
            StepName = stepName;
            Status = TestStatus.Pending;
        }
    }
    
    public class LightBDDResultListener : ITestResultListener
    {
        private readonly ILightBDDProgressNotifier _progressNotifier;
        
        public LightBDDResultListener(ILightBDDProgressNotifier progressNotifier)
        {
            _progressNotifier = progressNotifier;
        }
        
        public void OnTestCompleted(TestResult result)
        {
            if (result is LightBDDScenarioResult scenarioResult)
            {
                // Notify LightBDD about scenario completion
                _progressNotifier.NotifyScenarioResult(
                    scenarioResult.Metadata["FeatureDescription"].ToString(),
                    scenarioResult.Metadata["ScenarioTitle"].ToString(),
                    ConvertToLightBDDStatus(scenarioResult.Status),
                    scenarioResult.Duration,
                    scenarioResult.StepResults.Select(sr => new 
                    {
                        Name = sr.StepName,
                        Status = ConvertToLightBDDStatus(sr.Status),
                        Duration = sr.Duration,
                        Error = sr.Exception
                    }).ToList()
                );
            }
        }
        
        private object ConvertToLightBDDStatus(TestStatus status)
        {
            // Convert TUnit status to LightBDD status
            // Implementation details depend on LightBDD's API
            return null;
        }
    }
}

4. Integration Configuration

Finally, a configuration class would tie everything together:

namespace TUnit.Extensions.LightBDD
{
    public static class TUnitLightBDDExtensions
    {
        public static TUnitBuilder UseLightBDD(this TUnitBuilder builder)
        {
            // Register LightBDD test discoverer
            builder.Services.AddSingleton<ITestDiscoverer, LightBDDTestDiscoverer>();
            
            // Register LightBDD test runner
            builder.Services.AddSingleton<ITestRunner, LightBDDTestRunner>();
            
            // Register LightBDD progress notifier
            builder.Services.AddSingleton<ILightBDDProgressNotifier, LightBDDProgressNotifier>();
            
            // Register result listener
            builder.Services.AddSingleton<ITestResultListener, LightBDDResultListener>();
            
            return builder;
        }
    }
    
    public interface ILightBDDProgressNotifier
    {
        void NotifyScenarioResult(string featureDescription, string scenarioTitle, 
            object status, TimeSpan duration, IReadOnlyList<object> steps);
    }
    
    public class LightBDDProgressNotifier : ILightBDDProgressNotifier
    {
        // Implementation that integrates with LightBDD's progress notification system
        public void NotifyScenarioResult(string featureDescription, string scenarioTitle, 
            object status, TimeSpan duration, IReadOnlyList<object> steps)
        {
            // Send notifications to LightBDD's reporting system
        }
    }
}

Usage Example

With these implementations, the integration could be used like this:

// Configure TUnit to use LightBDD
var tunit = new TUnitBuilder()
    .UseLightBDD()
    .Build();

// Run tests
var results = await tunit.RunAsync(Assembly.GetExecutingAssembly());

// Example LightBDD test that would work with this integration
[FeatureDescription("Shopping Cart")]
public class ShoppingCartFeature : FeatureFixture
{
    [Scenario]
    public async Task Adding_product_to_cart()
    {
        await Runner.RunScenarioAsync(
            given => an_empty_cart(),
            when => a_product_is_added_to_cart(),
            then => the_cart_should_contain_one_item(),
            and => the_cart_total_should_be_updated()
        );
    }
    
    private Task an_empty_cart() => Task.CompletedTask;
    private Task a_product_is_added_to_cart() => Task.CompletedTask;
    private Task the_cart_should_contain_one_item() => Task.CompletedTask;
    private Task the_cart_total_should_be_updated() => Task.CompletedTask;
}

How TUnit Fulfills Suremaker's Requirements

1. Test Discovery

   • TUnit's existing reflection-based discovery is extended to recognize LightBDD's attributes
   • The LightBDDTestDiscoverer identifies scenario methods and feature classes
   • Step extraction allows for complete scenario structure discovery

2. Test Execution

   • The LightBDDTestRunner handles the specialized execution flow for LightBDD scenarios
   • LightBDD's Runner property is accessed to integrate with its step execution
   • Proper exception handling maintains test isolation

3. Test Result Capture

   • LightBDDScenarioResult extends TUnit's result model to capture step-level details
   • The LightBDDResultListener provides progress notifications to LightBDD's reporting system
   • Complete metadata enables rich reporting capabilities

This implementation approach maintains TUnit's extensible architecture while providing all the hooks needed for LightBDD integration, fulfilling Suremaker's three key requirements while staying true to both frameworks' design principles.

[thomhurst]

I think the only blocker or thing that would have to be different is:

ability to provide [Scenario] attribute and properly interpret it as scenario. NUnit does it this way: ScenarioAttribute

Because TUnit is source generated, the compiler looks for a well-known attribute, which is the [Test] attribute. So tests would have to stick with [Test] as opposed to [Scenario].

I personally don't think that's a problem, but obviously it's not my library and the author may want to keep things consistent with the other frameworks.

[dystopiandev]

Because TUnit is source generated, the compiler looks for a well-known attribute, which is the [Test] attribute. So tests would have to stick with [Test] as opposed to [Scenario].

@thomhurst Not my most sober thought but wouldn't it suffice to add a dummy extension in the TUnit namespace like...

class ScenarioAttribute : TestAttribute {}

Ok maybe not in the TUnit namespace since doing that from the LightBDD side would be invasive, but would discovery work with inheriting the attr like that though?

[thomhurst]

Nah the generator doesn't look through the inheritance structure. It just looks for a specific fully qualified type: TUnit.Core.TestAttribute.

I wonder if I could make the source generator be able to pluck a fully qualified type out of the project properties.

<TUnitTestAttribute>LightBDD.TUnit.ScenarioAttribute</TUnitTestAttribute>

[thomhurst]

LightBDD/LightBDD#376

[thomhurst]

This how now been released!