Skip to content

perf(knowledge): facts/by_category uses SMEMBERS+sorted() per request (no server-side SET pagination); legacy SCAN offset unstable #12426

Description

@mrveiss

Low-urgency follow-ups noted in PR #12424 (#12394) review — non-blocking, tracked for scale:

  1. _fetch_category_fact_ids loads ALL fact IDs per category via SMEMBERS + sorts O(N log N) every request. Fine today (only the sliced facts' CONTENT is fetched; matches the sibling knowledge_categories convention; default browse iterates just 3 KnowledgeCategory members). Becomes a per-request CPU/memory cost only if a single category grows to ~millions of facts — then it'd want server-side SET pagination (e.g. a sorted-set ZRANGE index).
  2. _get_facts_by_category_legacy (degraded no-index SCAN path) paginates over UNSORTED scan order → offset windows not stable across calls. Documented as 'good enough' for the fallback; only matters if that path is exercised interactively.
  3. UX: folder badges (categoryCounts) show 'loaded so far', not the new total_count — consider 'X of Y'.
    Refs perf(knowledge): paginate facts/by_category (unbounded list) #12394 perf(knowledge): facts/by_category browse list ships full_content (107KB/fact) → 3MB for 43 facts, unpaginated; detail endpoint already exists for lazy-load #12370.

Activity

  1. mrveiss commented on Jul 25, 2026

    @mrveiss
    OwnerAuthor

    Skipped in overnight autonomous run: low-priority perf follow-up explicitly 'fine today' (only a cost if a single category grows very large), and the fix approach (server-side SET pagination via sorted-set migration vs cursor cache) is a design choice, not an unambiguous fix. Left for prioritized/supervised work.

  2. modified the milestones: Backlog, v0.12.0 on Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions