Add ToStreamPageAsync and execute end cursors in EF paging by michaelstaib · Pull Request #4 · michaelstaib/graphql-platform

michaelstaib · 2026-09-27T01:33:19Z

This adds ToStreamPageAsync, which runs a paged EF Core query and hands back a StreamPage<T> as soon as the first row arrives. The remaining rows are yielded as the database returns them. Backward pages (last) never over-fetch: the query takes exactly N rows in reverse and the database re-sorts them into ascending order. ToPageAsync and ToBatchPageAsync now execute end cursors: an end cursor always forces a fresh total count, and one that points before the first page returns an empty page. To keep all three methods in sync, the argument validation and slicing move into a shared internal query composer; existing SQL snapshots are unchanged. Reviewers should check the backward-flag and lifetime-disposal paths in ToStreamPageAsync, where a few edge cases are still open with the planner.

await using var page = await context.Products
    .OrderBy(p => p.Name).ThenBy(p => p.Id)
    .ToStreamPageAsync(new PagingArguments { First = 10 }, lifetime: scope);

await foreach (var product in page) { }   // rows arrive as the reader produces them

Closed tickets: hc-fork-1-o47.1 (968fcd9), hc-fork-1-o47.2 (f8b1293, 7f7e25d, d0fa596), hc-fork-1-o47.3 (e021e28), hc-fork-1-o47.5 (no code change; EXPLAIN shows the key filter already sits inside the ROW_NUMBER window input).

Stack: sits on #3 (batch pump edge cases, mst/primitives-pump-fixes).

🤖 Generated with Claude Code

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved lifetime-disposal paths and end-cursor validation ordering require correction.

Review effort: Lite
Findings: 2 · 2

Open (4)

Dispose lifetime on validation and pre-pump failures · New
Dispose pump resources when priming fails · New
Centralize end-cursor validation exception creation · New
Centralize validation exceptions through EntityFramework ThrowHelper · New

What changed in this PR

Adds EF Core streaming pagination, backward-page handling, and end-cursor support across paging APIs.

Changes:

Introduces ToStreamPageAsync with forward and backward streaming.
Centralizes paging composition and end-cursor slicing.
Adds comprehensive paging tests and snapshots.

File	Description
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/StreamPagingHelperTests.cs`	Streaming pagination tests.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/RelativeCursorTests.cs`	End-cursor and batch tests.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Rows_Only_NET9_0.snap`	NET9 streaming snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Rows_Only_NET11_0.snap`	NET11 streaming snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Rows_Only_NET10_0.snap`	NET10 streaming snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Forward_First_After_NET9_0.snap`	NET9 forward-page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Forward_First_After_NET11_0.snap`	NET11 forward-page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Forward_First_After_NET10_0.snap`	NET10 forward-page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_End_Cursor_25_10_NET9_0.snap`	NET9 end-cursor snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_End_Cursor_25_10_NET11_0.snap`	NET11 end-cursor snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_End_Cursor_25_10_NET10_0.snap`	NET10 end-cursor snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Empty_Result_With_Count_NET9_0.snap`	NET9 empty-result snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Empty_Result_With_Count_NET11_0.snap`	NET11 empty-result snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Empty_Result_With_Count_NET10_0.snap`	NET10 empty-result snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Count_Only_NET9_0.snap`	NET9 count-only snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Count_Only_NET11_0.snap`	NET11 count-only snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Count_Only_NET10_0.snap`	NET10 count-only snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Backward_Last_Before_NET9_0.snap`	NET9 backward-page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Backward_Last_Before_NET11_0.snap`	NET11 backward-page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/StreamPagingHelperTests.Fetch_Backward_Last_Before_NET10_0.snap`	NET10 backward-page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/RelativeCursorTests.Fetch_End_Cursor_Two_Pages_Before_Last.snap`	End-cursor page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/RelativeCursorTests.Fetch_End_Cursor_One_Page_Before_Last.snap`	End-cursor page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/RelativeCursorTests.Fetch_End_Cursor_One_Page_Before_Last_Forces_Fresh_Count.snap`	Fresh-count snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/RelativeCursorTests.Fetch_End_Cursor_Offset_Before_First_Page_Returns_Empty_Page.snap`	Empty out-of-range snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/RelativeCursorTests.Fetch_End_Cursor_Last_Page_Smaller_Than_RequestedSize.snap`	Short final-page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/RelativeCursorTests.Fetch_End_Cursor_Last_Page_Not_Evenly_Divisible.snap`	Uneven final-page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/RelativeCursorTests.Fetch_End_Cursor_Last_Page_Evenly_Divisible.snap`	Even final-page snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/RelativeCursorTests.Fetch_End_Cursor_Last_Page_Empty_Dataset.snap`	Empty-dataset snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/RelativeCursorTests.BatchFetch_End_Cursor_Trims_Per_Key_Total.snap`	Batch totals snapshot.
`src/GreenDonut/test/GreenDonut.Data.EntityFramework.Tests/__snapshots__/RelativeCursorTests.BatchFetch_End_Cursor_Offset_Before_First_Page_Returns_Empty_Pages.snap`	Batch empty-page snapshot.
`src/GreenDonut/src/GreenDonut.Data.EntityFramework/Internal/PagingQueryComposer.cs`	Shared paging composition logic.
`src/GreenDonut/src/GreenDonut.Data.EntityFramework/Extensions/StreamPagingQueryableExtensions.cs`	Streaming page implementation.
`src/GreenDonut/src/GreenDonut.Data.EntityFramework/Extensions/PagingQueryableExtensions.cs`	End-cursor support.
`src/GreenDonut/src/GreenDonut.Data.EntityFramework/Expressions/OriginalOrderReapplier.cs`	Restores backward-page ordering.
`src/GreenDonut/src/GreenDonut.Data.EntityFramework/Expressions/ExpressionHelpers.cs`	Batch end-cursor slicing and validation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate end-cursor issues affect batch slicing, range handling, metadata, and offset overflow.

Review effort: Lite
Findings: 3

Open (3)

Compute negative end-cursor offsets per group · New
Apply before-first checks using each group's total · New
Prevent overflow for minimum end-cursor offsets · New

Resolved since last review (4)

Dispose pump resources when priming fails
Dispose lifetime on validation and pre-pump failures
Centralize validation exceptions through EntityFramework ThrowHelper
Centralize end-cursor validation exception creation

michaelstaib · 2026-09-28T08:11:28Z

+            var cachedTotal = cursor!.TotalCount!.Value;
+            var pagesBeforeLast = -cursor.Offset!.Value;
+
+            if (pagesBeforeLast > 0)
+            {
+                var remainder = cachedTotal % requestedCount == 0
+                    ? requestedCount
+                    : cachedTotal % requestedCount;
+                var skip = remainder + (pagesBeforeLast - 1) * requestedCount;


Agree. Superseding the uniform-cached-total rule: batch end cursors with a negative offset align PER KEY — the group expression fetches 2N rows from the end after Skip((k−1)×N) (a window that always contains the key's page), then each key is trimmed by its own freshly counted total. Follow-up PR (task hc-fork-1-o47.13).

michaelstaib · 2026-09-28T08:11:29Z

+            if (pagesBeforeLast > 0
+                && (int)Math.Ceiling(endCursor.TotalCount!.Value / (double)arguments.Last.Value) - pagesBeforeLast < 1)
+            {
+                foreach (var (key, count) in counts!)
+                {
+                    map.Add(key, CreateEndCursorPage<TValue>([], keys, false, false, 1, arguments.Last.Value, count));
+                }
+
+                return map;


Agree. The out-of-range test moves per key: index_k = ceil(total_k / N) − k, and only a key whose index falls below 1 gets an empty page; the other keys return their pages. Same follow-up as the per-key skip (task hc-fork-1-o47.13).

michaelstaib · 2026-09-28T08:08:34Z

+                offset = cursor.Offset!.Value;
+                var cachedTotal = cursor.TotalCount!.Value;
+                var pagesBeforeLast = -offset;


Agree. Offsets are validated so that negation and the skip arithmetic cannot overflow (int.MinValue and any offset whose skip does not fit an int are rejected with ArgumentException; the parser rejects int.MinValue). Follow-up PR (task hc-fork-1-o47.10).

…geAsync

Copilot AI lite review requested due to automatic review settings September 27, 2026 01:33

Copilot started reviewing on behalf of michaelstaib September 27, 2026 01:34 View session

Copilot AI reviewed Sep 27, 2026

View reviewed changes

michaelstaib mentioned this pull request Sep 27, 2026

Add StreamPage DataLoader extensions and hash IncludeItems in branch keys #5

Merged

michaelstaib added this pull request to stack #7 September 27, 2026 10:23

michaelstaib force-pushed the mst/ef-paging-streampage branch from e021e28 to 34528a6 Compare September 27, 2026 20:58

github-actions Bot added the 🌶️ green donut label Sep 27, 2026

michaelstaib force-pushed the mst/ef-paging-streampage branch from 34528a6 to 5e14e55 Compare September 27, 2026 21:40

Base automatically changed from mst/primitives-pump-fixes to mst/stream-v2 September 27, 2026 22:02

michaelstaib force-pushed the mst/ef-paging-streampage branch from 5e14e55 to 2e14ead Compare September 27, 2026 22:02

michaelstaib requested a lite review from Copilot September 27, 2026 22:03

Copilot started reviewing on behalf of michaelstaib September 27, 2026 22:03 View session

Copilot AI reviewed Sep 27, 2026

View reviewed changes

michaelstaib added 5 commits September 28, 2026 10:08

Extract shared paging query composition from ToPageAsync

f27dd46

Execute end cursors for last-page paging in ToPageAsync and ToBatchPa…

a9e82ea

…geAsync

Force the total count for every end cursor page

6badbf9

Return an empty page for end cursors that point before the first page

fe5d542

Implement ToStreamPageAsync

e770f85

michaelstaib force-pushed the mst/ef-paging-streampage branch from 2e14ead to e770f85 Compare September 28, 2026 08:08

michaelstaib merged commit 7ed8ae1 into mst/stream-v2 Sep 28, 2026
1 check passed

Navigation Menu

Repository navigation

Add ToStreamPageAsync and execute end cursors in EF paging - #4

Add ToStreamPageAsync and execute end cursors in EF paging#4
michaelstaib merged 5 commits into
mst/stream-v2from
mst/ef-paging-streampage

michaelstaib commented Sep 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

michaelstaib Sep 28, 2026

Uh oh!

michaelstaib Sep 28, 2026

Uh oh!

michaelstaib Sep 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

michaelstaib commented Sep 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

michaelstaib Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

michaelstaib Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

michaelstaib Sep 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants