Skip to content

Conversation

Copy link

Copilot AI commented Jul 22, 2025

This PR implements IAsyncEnumerable<T> support for the QueryAsync methods in Azure Storage Table, enabling efficient streaming of large datasets without loading all results into memory at once.

New Features

🚀 New API Methods

  • QueryAsyncEnumerable<T>(string partitionKey, int maxItems = 0)
  • QueryAsyncEnumerable<T>(string partitionKey, IEnumerable<QueryFilter> queryFilters, int maxItems = 0)
  • QueryAsyncEnumerable<T>(int maxItems = 0)
  • Query<T>().AsAsyncEnumerable() for fluent API integration

💡 Usage Examples

// Stream large datasets efficiently
await foreach (var user in storageContext.QueryAsyncEnumerable<UserModel>("users", 1000))
{
    Console.WriteLine($"Processing user: {user.Name}");
    await ProcessUserAsync(user);
}

// With cancellation support
using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(5));
await foreach (var product in storageContext
    .Query<Product>()
    .InPartition("electronics")
    .LimitTo(500)
    .AsAsyncEnumerable()
    .WithCancellation(cts.Token))
{
    // Process each product individually
}

// Memory-efficient processing of millions of records
await foreach (var order in storageContext.QueryAsyncEnumerable<Order>(maxItems: 1000000))
{
    totalValue += order.Amount;
    processedCount++;
    
    if (processedCount % 10000 == 0)
    {
        Console.WriteLine($"Processed {processedCount} orders...");
    }
}

Key Benefits

  • Memory Efficient: Items are streamed one-by-one instead of loading entire result sets
  • Performance: Lazy evaluation with early termination support
  • Cancellation: Built-in CancellationToken support via WithCancellation()
  • Developer Experience: Natural await foreach syntax
  • Scalability: Process datasets of any size with constant memory usage

Framework Compatibility

IAsyncEnumerable support is available on:

  • .NET Standard 2.1+
  • .NET Core 3.0+
  • .NET 5.0+
  • .NET 8.0+

For older frameworks (.NET Standard 2.0, .NET Framework 4.8), the library automatically includes the Microsoft.Bcl.AsyncInterfaces package for compatibility.

Implementation Details

  • Conditional Compilation: Uses #if NETSTANDARD2_1_OR_GREATER || NET5_0_OR_GREATER to enable features only where supported
  • Azure Integration: Leverages existing Azure Table Storage pagination with AsPages().GetAsyncEnumerator()
  • Backward Compatible: All existing QueryAsync methods continue to work unchanged
  • Resource Management: Proper async disposal with IAsyncDisposable implementation

Technical Changes

  • Added Microsoft.Bcl.AsyncInterfaces package dependency for older target frameworks
  • Created StorageContextQueryIAsyncEnumerable<T> implementing IAsyncEnumerable<T>
  • Created StorageContextQueryIAsyncEnumerator<T> implementing IAsyncEnumerator<T>
  • Extended IStorageContext and query interfaces with async enumerable methods
  • Integrated with existing query pipeline and filtering system

Testing

  • ✅ Builds successfully on all target frameworks (netstandard2.0, netstandard2.1, net48, net8.0)
  • ✅ Compilation and runtime functionality verified
  • ✅ Backward compatibility maintained
  • ✅ Integration with existing features (filters, pagination, delegates) confirmed

This enhancement provides a powerful new way to work with large datasets in Azure Table Storage while maintaining full backward compatibility with existing code.

Fixes #15.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: petero-dk <2478689+petero-dk@users.noreply.github.com>
Copilot AI changed the title [WIP] GetAsyncEnumerator support Add IAsyncEnumerable support for QueryAsync methods Jul 22, 2025
Copilot AI requested a review from petero-dk July 22, 2025 18:58
@petero-dk petero-dk changed the base branch from release/v6.5 to release/v7.0 July 24, 2025 18:50
@petero-dk petero-dk marked this pull request as ready for review July 24, 2025 18:50
@petero-dk petero-dk merged commit df00f00 into release/v7.0 Jul 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GetAsyncEnumerator support

2 participants