Skip to content

fix(knowledge): KB upload loses page provenance — facts cannot cite a source page #13894

Closed

Description

@mrveiss

Part of #13892 — Wave 2. Depends on #13893 (extractor consolidation).

Summary

The live KB upload path joins pages with a bare newline and no page marker, so
nothing downstream can say which page a retrieved fact came from. Citable
retrieval is impossible for uploaded PDFs.

autobot-backend/api/knowledge.py:1065:

return "\n".join(page.extract_text() or "" for page in pdf_reader.pages)

Three conventions, and the live path has none

Location Marker
api/knowledge.py:1065 — live upload none
media/document/pipeline.py:104 — live none
knowledge/connectors/content_extraction.py:39 — live ## Page N
utils/document_parser.py:124 — orphan --- Page N ---
utils/document_extractors.py:126 — orphan none

#13893 picks the surviving convention; this issue carries it through to retrieval.

Approach

Carry the page number as chunk metadata, not inline text. An inline ## Page 7
marker leaks into the embedding and pollutes similarity — the marker is structural,
not semantic.

Constraints:

  • The chunker must not split across a page boundary in a way that orphans the
    attribution, or a chunk ends up spanning two pages with one page number.
  • Applies to the vectorization path, not only extraction — metadata has to survive
    all the way to the retrieved result.
  • A chunk that genuinely spans pages should record a range, not silently pick one.

Acceptance criteria

  • Uploaded PDF chunks carry a page number (or range) in metadata
  • Page markers are not embedded in the text sent to the embedding model
  • A KB search result can report the source page for an uploaded PDF
  • Drive/OneDrive connector ingestion carries the same metadata
  • Test asserts a retrieved fact resolves to the correct page of a multi-page fixture

Note

The external long-horizon OCR approach that prompted this audit is worse here —
it emits an unanchored markdown blob with no page anchors at all. This takes the
idea by inverting it.

Activity

  1. github-actions commented on Aug 15, 2026

    @github-actions
    Contributor

    PR #14239 (merged to Dev_new_gui) references this issue with a close keyword.

    feat(knowledge): carry page provenance as metadata, not marker text (#13894)

    If this issue is fully resolved, close it manually. If work remains, no action is needed.

  2. mrveiss commented on Aug 15, 2026

    @mrveiss
    OwnerAuthor

    Landed in Dev_new_gui as 852fd4c36 (PR #14239).

    Verified in base, not from the merge result:

    f6bf3847af33271e93a97047c61853df27c15c59
    fix(skills): an unreadable conversation stops the pass instead of being reported as distilled (#14077) (#14247)
    

    Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions