Skip to content

row paging on an aggregate-backed xorq session re-executes the full aggregation per 300-row window #923

Description

@paddymul

Problem

handle_infinite_request_xorq (xorq_loading.py:421) serves each scroll window by executing the session's expression through window_to_parquet(filtered_expr, start, end, sort_col, ascending). For scan-backed expressions, limit/offset pushdown makes this near-free. For aggregate-backed expressions the entire aggregation re-runs for every 300-row window, and a column sort re-runs the aggregation plus a full sort. _expr_count is cached per expression (#889); the window execution is not.

Evidence

  • 2026-06-10 parking_ticket_analysis replay (0.14.17): scan-backed entries page in 5–12ms; aggregate-backed entries (habitual_speeder_stats: group_by over 131.7M rows → 4.5M; tickets_per_day: 3 aggregations + 2 joins) take 1.35–1.75s per window, every window.
  • tallyman perf regression suite (perf(suite): report-only perf regression suite + CI workflow tallyman#36; synthetic 4M rows → 1.4M-group count/min/max aggregate, 4-core CI runner): first page 0.26s vs 0.05s for the scan-backed control over the same data.

Impact

Scrolling an aggregate-backed table costs one full aggregation per viewport, and sorting costs aggregation + sort per window — ~1.4s added to every scroll/sort interaction on the parking catalog, scaling with input size, on entries whose final results are small (5K–4.5M rows).

Suggested fix

Serve windows from a materialized result when the source is chained. #916 already classifies exactly these op trees (filter / aggregate / join / distinct / sort) to decide stat-source materialization; reusing that materialized table for row serving — or accepting an embedder-provided result parquet (buckaroo-data/tallyman#31 sketches the protocol) — would make paging O(window) instead of O(aggregation).

Context

Identified during the 2026-06-10 parking_ticket_analysis replay and re-measured by tallyman's perf suite. Present on current main (handle_infinite_request_xorqwindow_to_parquet); distinct from #899 (repeat /load_expr cost) and untouched by #901/#913/#916, which address the load/stat side.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions