Repository navigation
Conversation
…16773) detect_format verified pdf and docx by magic bytes; the other six office formats were routed on their extension alone, by document_parser's extension -> parser dict and document_extractors' suffix routing. All seven share the PK prefix, so a renamed .xlsx/.pptx/.odt/.ods/.odp/.odg was handed to the wrong parser or refused outright even though its content was fully parseable. - media/document/zip_formats.py: reads the archive's own members -- the OOXML part only that type carries, or the ODF mimetype member ODF stores first. No new dependency: zipfile is stdlib and detect_format already unzips docx this way. Reads the central directory and at most one capped member, never the archive. - detect_format returns the verified format; the existing marker sniff still covers a truncated upload whose central directory has not arrived. - DocumentParser leads with the verified format and still tries the name after it, so a mismatch alone never fails an extraction, mirroring pdf/docx today. - DocumentExtractor routes on the verified format when it disagrees with the name. Fixtures are built with zipfile in-test, so what each assertion depends on is visible rather than hidden in a committed binary. Refs docs/research/document-to-markdown-conversion-pipeline.md ("What We Can Adopt" #1) Closes #16773
…16775) ALLOWED_EXTENSIONS omitted .xlsx/.pptx/.odt/.ods/.odp even though DocumentParser has carried working parsers for them all along, so a user could not upload a file the backend could already read by another path. Owner decision on the ingestion review: every format in it is to be handled. - api/knowledge_office_upload.py: the routing, and no new extraction logic. - Dispatch resolves the format from content first (#16773), so widening the allowlist does not reintroduce the extension-trust that issue just removed. A renamed .xlsx reaches the workbook parser, not the one its name claims. - utils/document_parser.parse_document_text: the sync entry point for a caller already off the event loop. The upload route runs inside asyncio.to_thread, so it needs the same dispatch without a second thread hop, and without a second copy of the parser table — which is how these two routes drifted before. Left out deliberately: .doc and .ppt are OLE2, and python-docx/python-pptx read only the OOXML ones, so advertising them would accept a valid file and then call it corrupt (#16786, an owner call). .odg is left out too — a drawing carries almost no extractable text and this endpoint stores flattened text only. api/knowledge.py is at its size ceiling, so its edits are net-zero: the routing lives in the new module, and three added lines are paid for by rewrapping three comments. Closes #16775 Refs #16772
…GUI upload path (#16785) DocumentExtractor.SUPPORTED_FORMATS had no entry for .csv, .json or .html, so extract_from_file() raised "Unsupported file type" for all three while api.knowledge's GUI upload path (_extract_file_content) ingested them deliberately: CSV as plain text, JSON re-serialised with indent=2 so its structure survives embedding, HTML through _sanitize_html_content. Same file, two divergent outcomes depending on whether it arrived through the upload form or a connector/directory walk. - .csv joins the existing "text" category -- it's handled as plain text on both routes, nothing new to write. - New extract_from_json(): mirrors the upload path's structure exactly, including its edge case (a JSONDecodeError falls back to raw text, but a non-UTF-8 file still raises -- the fallback decode only runs after a JSONDecodeError, not a UnicodeDecodeError). - New extract_from_html(): imports api.knowledge._sanitize_html_content at call time and calls it directly -- not a second sanitiser. Lazy import because this module is reusable by any component and a FastAPI router module is not a dependency a plain-text extraction should carry for callers that never touch HTML. - get_supported_extensions()/is_supported_format() pick up all three automatically (they already iterate SUPPORTED_FORMATS generically), so directory discovery (knowledge/documents.py) stops silently skipping them -- docstring there updated to match what's actually handled now. Tested by driving both routes (DocumentExtractor and api.knowledge._extract_file_content) on identical bytes for all three formats and asserting they produce the same text, since two routes agreeing is the fix -- not merely "no longer raises". Refs #16785
…-secrets HexHighEntropyString
|
Warning Review limit reachedNext included review available in 17 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Repository: mrveiss/AutoBot-AI/.coderabbit.yaml Review profile: ASSERTIVE Plan: Advanced Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (13)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
✅ SSOT Configuration Compliance: Passing🎉 No new hardcoded values of either class — Known backlog in |
Thinking Path
The owner's new same-file rule groups #16783, #16788 and #16790 because they touch the document-parsing/knowledge-upload surface together. #16788 is stacked on #16783 (
base: issue-16773-zip-format-verify, notmain) — content verification has to land before the upload allowlist widens on top of it, so #16783's head is merged first, then #16788's head, exactly as instructed. #16790 is independent (basemain) and merges last.This branch is built entirely through the REST API: created from
main, then each approved head merged in withPOST /repos/.../merges, in that order.Named
vehicle-v090-2026-09-18-docparser, flat with no slash.What Changed
Nothing new. Three already-reviewed heads, each merged cleanly with no conflicts:
No excluded members — all three passed the ledger gate (approve at current head, no STALE, no failing check, open) and merged without conflict, including the deliberate #16783-before-#16788 ordering.
Verification
Each member carries a ledger review verdict pinned to the exact head merged here — the table above, all non-stale at merge time. #16788's
closingIssuesReferencescomes back empty from the GitHub API only because its declared base is #16783's branch rather thanmain; its own PR body statesCloses #16775explicitly, which is what this vehicle carries forward.What this PR's own CI must establish is that the union holds: no conflict between members, no ratchet baseline moved,
api/knowledge.pystill at its size ceiling.Model Used
Claude Sonnet 5 (coordinator session, vehicle build only — no new code). Members authored and reviewed by their own PR sessions per the ledger verdicts above.
Closes #16773
Closes #16775
Closes #16785
Lint fix commit
9884eca -- lint-only, produced by black/isort/flake8 at the pinned versions, needs non-author review.
Files touched:
autobot-backend/api/knowledge.py-- flake8 F401 / autoflake: unused function-localimport osin_extract_file_contentremoved (left unused by KB upload allowlist omits already-supported office/ODF formats #16775)repo_tests/python_file_size_ratchet_baseline.py,scripts/python_file_size_known_large.py--knowledge.pyceiling lowered 3414 -> 3413 in both, to match the removed line (the ratchet only turns down)