Skip to content

chore(vehicle): land document-parser stack — 3 approved PRs (#16783, #16788, #16790) - #17063

Closed
mrveiss wants to merge 27 commits into
mainfrom
vehicle-v090-2026-09-18-docparser
Closed

mrveiss wants to merge 27 commits into
mainfrom
vehicle-v090-2026-09-18-docparser

Conversation

@mrveiss

@mrveiss mrveiss commented Sep 18, 2026 •

Copy link
Copy Markdown
Owner

Thinking Path

The owner's new same-file rule groups #16783, #16788 and #16790 because they touch the document-parsing/knowledge-upload surface together. #16788 is stacked on #16783 (base: issue-16773-zip-format-verify, not main) — content verification has to land before the upload allowlist widens on top of it, so #16783's head is merged first, then #16788's head, exactly as instructed. #16790 is independent (base main) and merges last.

This branch is built entirely through the REST API: created from main, then each approved head merged in with POST /repos/.../merges, in that order.

Named vehicle-v090-2026-09-18-docparser, flat with no slash.

What Changed

Nothing new. Three already-reviewed heads, each merged cleanly with no conflicts:

PR Head Delivers Ledger verdict
#16783 ca95302 #16773 — verify ZIP-based office/ODF formats by content, not extension approve@ca95302a2
#16788 cfb0d49 #16775 — KB upload accepts .xlsx/.pptx/.odt/.ods/.odp, stacked on #16783 approve@cfb0d4932
#16790 c5f51c0 #16785 — DocumentExtractor handles .csv/.json/.html like the GUI upload path approve@c5f51c052

No excluded members — all three passed the ledger gate (approve at current head, no STALE, no failing check, open) and merged without conflict, including the deliberate #16783-before-#16788 ordering.

Verification

Each member carries a ledger review verdict pinned to the exact head merged here — the table above, all non-stale at merge time. #16788's closingIssuesReferences comes back empty from the GitHub API only because its declared base is #16783's branch rather than main; its own PR body states Closes #16775 explicitly, which is what this vehicle carries forward.

What this PR's own CI must establish is that the union holds: no conflict between members, no ratchet baseline moved, api/knowledge.py still at its size ceiling.

Model Used

Claude Sonnet 5 (coordinator session, vehicle build only — no new code). Members authored and reviewed by their own PR sessions per the ledger verdicts above.

Closes #16773
Closes #16775
Closes #16785

Lint fix commit

9884eca -- lint-only, produced by black/isort/flake8 at the pinned versions, needs non-author review.

Files touched:

  • autobot-backend/api/knowledge.py -- flake8 F401 / autoflake: unused function-local import os in _extract_file_content removed (left unused by KB upload allowlist omits already-supported office/ODF formats #16775)
  • repo_tests/python_file_size_ratchet_baseline.py, scripts/python_file_size_known_large.py -- knowledge.py ceiling lowered 3414 -> 3413 in both, to match the removed line (the ratchet only turns down)

mrveiss and others added 26 commits September 16, 2026 15:54
…16773)

detect_format verified pdf and docx by magic bytes; the other six office formats
were routed on their extension alone, by document_parser's extension -> parser
dict and document_extractors' suffix routing. All seven share the PK prefix, so a
renamed .xlsx/.pptx/.odt/.ods/.odp/.odg was handed to the wrong parser or refused
outright even though its content was fully parseable.

- media/document/zip_formats.py: reads the archive's own members -- the OOXML part
  only that type carries, or the ODF mimetype member ODF stores first. No new
  dependency: zipfile is stdlib and detect_format already unzips docx this way.
  Reads the central directory and at most one capped member, never the archive.
- detect_format returns the verified format; the existing marker sniff still covers
  a truncated upload whose central directory has not arrived.
- DocumentParser leads with the verified format and still tries the name after it,
  so a mismatch alone never fails an extraction, mirroring pdf/docx today.
- DocumentExtractor routes on the verified format when it disagrees with the name.

Fixtures are built with zipfile in-test, so what each assertion depends on is
visible rather than hidden in a committed binary.

Refs docs/research/document-to-markdown-conversion-pipeline.md ("What We Can Adopt" #1)
Closes #16773
…16775)

ALLOWED_EXTENSIONS omitted .xlsx/.pptx/.odt/.ods/.odp even though DocumentParser
has carried working parsers for them all along, so a user could not upload a file
the backend could already read by another path. Owner decision on the ingestion
review: every format in it is to be handled.

- api/knowledge_office_upload.py: the routing, and no new extraction logic.
- Dispatch resolves the format from content first (#16773), so widening the
  allowlist does not reintroduce the extension-trust that issue just removed.
  A renamed .xlsx reaches the workbook parser, not the one its name claims.
- utils/document_parser.parse_document_text: the sync entry point for a caller
  already off the event loop. The upload route runs inside asyncio.to_thread, so
  it needs the same dispatch without a second thread hop, and without a second
  copy of the parser table — which is how these two routes drifted before.

Left out deliberately: .doc and .ppt are OLE2, and python-docx/python-pptx read
only the OOXML ones, so advertising them would accept a valid file and then call
it corrupt (#16786, an owner call). .odg is left out too — a drawing carries
almost no extractable text and this endpoint stores flattened text only.

api/knowledge.py is at its size ceiling, so its edits are net-zero: the routing
lives in the new module, and three added lines are paid for by rewrapping three
comments.

Closes #16775
Refs #16772
…GUI upload path (#16785)

DocumentExtractor.SUPPORTED_FORMATS had no entry for .csv, .json or .html,
so extract_from_file() raised "Unsupported file type" for all three while
api.knowledge's GUI upload path (_extract_file_content) ingested them
deliberately: CSV as plain text, JSON re-serialised with indent=2 so its
structure survives embedding, HTML through _sanitize_html_content. Same
file, two divergent outcomes depending on whether it arrived through the
upload form or a connector/directory walk.

- .csv joins the existing "text" category -- it's handled as plain text on
  both routes, nothing new to write.
- New extract_from_json(): mirrors the upload path's structure exactly,
  including its edge case (a JSONDecodeError falls back to raw text, but a
  non-UTF-8 file still raises -- the fallback decode only runs after a
  JSONDecodeError, not a UnicodeDecodeError).
- New extract_from_html(): imports api.knowledge._sanitize_html_content at
  call time and calls it directly -- not a second sanitiser. Lazy import
  because this module is reusable by any component and a FastAPI router
  module is not a dependency a plain-text extraction should carry for
  callers that never touch HTML.
- get_supported_extensions()/is_supported_format() pick up all three
  automatically (they already iterate SUPPORTED_FORMATS generically), so
  directory discovery (knowledge/documents.py) stops silently skipping
  them -- docstring there updated to match what's actually handled now.

Tested by driving both routes (DocumentExtractor and
api.knowledge._extract_file_content) on identical bytes for all three
formats and asserting they produce the same text, since two routes
agreeing is the fix -- not merely "no longer raises".

Refs #16785
…ch (#16773)

main moved two commits (#16597, #16761) while CI ran, so the green run described
a base that had moved. No conflicts; this branch touches no size-ratcheted file.
#16788 is stacked on this branch and follows it rather than being updated itself.
@mrveiss mrveiss added this to the v0.9.0 milestone Sep 18, 2026
@coderabbitai

coderabbitai Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 17 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: mrveiss/AutoBot-AI/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 17d8ed61-d958-4c3e-b818-5e58afcd514f

📥 Commits

Reviewing files that changed from the base of the PR and between 0e9b3a8 and 9884eca.

⛔ Files ignored due to path filters (2)
  • repo_tests/python_file_size_ratchet_baseline.py is excluded by !repo_tests/python_file_size_ratchet_baseline.py
  • scripts/python_file_size_known_large.py is excluded by !scripts/python_file_size_known_large.py
📒 Files selected for processing (13)
  • .session/HANDOFF-issue-16784-16785-document-format-gaps.md
  • autobot-backend/api/knowledge.py
  • autobot-backend/api/knowledge_office_upload.py
  • autobot-backend/api/knowledge_office_upload_16775_test.py
  • autobot-backend/knowledge/documents.py
  • autobot-backend/media/document/extraction.py
  • autobot-backend/media/document/zip_formats.py
  • autobot-backend/media/document/zip_formats_16773_test.py
  • autobot-backend/utils/document_dispatch_16773_test.py
  • autobot-backend/utils/document_extractors.py
  • autobot-backend/utils/document_extractors_16785_test.py
  • autobot-backend/utils/document_parser.py
  • changelog/unreleased/16785-document-extractor-csv-json-html.md

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@mrveiss mrveiss added the land-next Landing set: CI capacity goes to these PRs first (owner direction 2026-09-18, #15397) label Sep 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Notice: 29 open PRs — past the runaway threshold (25)

There is no PR queue limit, and this is not a request to defer this PR. Work proceeds one issue at a time without a cap on open PRs; review capacity is the constraint.

This notice only means the count is high enough to be worth a glance for a runaway — something opening PRs in a loop, or a merge pipeline that has stalled so nothing is draining.

Currently open:

If the queue is draining normally, ignore this. Otherwise:

  1. Check whether CI is dispatching at all — see the ci-dispatch-watchdog status on these PRs
  2. Merge the ones whose CI has finished and review has passed: gh pr merge <number> --squash --delete-branch
  3. Look for a loop opening near-identical PRs

Warn-only runaway detector — .github/workflows/pr-queue-gate.yml. It never blocks a merge.

@github-actions

Copy link
Copy Markdown
Contributor

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.

@mrveiss

mrveiss commented Sep 18, 2026

Copy link
Copy Markdown
Owner Author

Superseded by #17077, a combined vehicle built from current main. It carries this vehicle's approved members at their current approved heads. The same-file rule puts #16895 and #16788 into one PR because both touch api/knowledge.py. The branch is kept, not deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

land-next Landing set: CI capacity goes to these PRs first (owner direction 2026-09-18, #15397)

Projects

None yet

1 participant