Skip to content

feat(query_client): add pagination support for records queries#1679

Draft
wingding12 wants to merge 3 commits intopydantic:mainfrom
wingding12:support-pagination-query-client-1608
Draft

feat(query_client): add pagination support for records queries#1679
wingding12 wants to merge 3 commits intopydantic:mainfrom
wingding12:support-pagination-query-client-1608

Conversation

@wingding12
Copy link

Closes #1608

Summary

Adds built-in pagination support to the Logfire query client to enable downloading more than the 10,000 row API limit.

Changes

  • Add iter_paginated_records() method to both LogfireQueryClient and AsyncLogfireQueryClient
  • Uses cursor-based pagination with (start_timestamp, trace_id, span_id) by default
  • Supports use_created_at=True for paginating over recent data where new rows may be inserted (uses created_at, trace_id, span_id, kind)
  • Add PaginationCursor TypedDict and MAX_QUERY_LIMIT constant
  • Add documentation in the query API guide
  • Add tests with VCR cassettes for both sync and async clients

Usage

with LogfireQueryClient(read_token=token) as client:
    for rows, next_cursor in client.iter_paginated_records(
        select='*',
        where="level >= 'error'",
        page_size=1000,
    ):
        process(rows)
        if next_cursor is None:
            break

…tic#1608)

Add iter_paginated_records() to both LogfireQueryClient and
AsyncLogfireQueryClient to enable downloading more than the 10,000 row
API limit. Uses cursor-based pagination with (start_timestamp, trace_id,
span_id) or (created_at, trace_id, span_id, kind) when use_created_at=True.

Implements pagination guidelines from maintainer discussion in the issue.
@codecov
Copy link

codecov bot commented Jan 31, 2026

Codecov Report

❌ Patch coverage is 91.30435% with 6 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
logfire/experimental/query_client.py 91.30% 2 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

…ponse path

- Unit tests for _build_paginated_records_sql (use_created_at, where+cursor, cursor only)
- Unit tests for _extract_cursor_from_row (use_created_at, missing keys returns None)
- Integration tests for empty response handling in iter_paginated_records
- Add VCR cassettes for empty response tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support pagination in logfire query client

1 participant