feat(knowledge): the batch document extractor rejects .csv, .json and .html that the upload path ingests #16785

Activity

1. mrveiss

   added

   enhancementNew feature or requestNew feature or request

   priority: medium

   backend

   area: document-ingestWave 5 · cluster K — Document ingest & OCRWave 5 · cluster K — Document ingest & OCR

   on Sep 16, 2026
2. mrveiss

   added a parent issue on Sep 16, 2026

   • Document ingestion hardening: content verification for office formats + zip ingestion #16772
3. mrveiss

   added 2 commits that reference this issue on Sep 16, 2026

   docs(changelog): add fragment for #16785 (DocumentExtractor csv/json/…

   ...

   327d092

   chore(session): write end-of-session handoff for issue-16784-16785-do…

   ...

   93c5155
4. mrveiss

   added this to the v0.9.0 milestone on Sep 17, 2026
5. mrveiss commented on Sep 19, 2026

   

   mrveiss

   on Sep 19, 2026

   OwnerAuthor

   More actions

   Closing with evidence — all four acceptance criteria verified against the actual merged code on origin/main at 928573e3bb (merge of #17077, PR #16790), not the PR body's claims.

   AC1 — .csv/.json/.html handled by extract_from_file, reusing the GUI path's treatment rather than decoding all three as plain text
   autobot-backend/utils/document_extractors.py:94-111 — SUPPORTED_FORMATS["text"] now includes .csv (line 99, decoded as text like .txt/.md), plus new "json": [".json"] (line 110) and "html": [".html"] (line 111) groups. Dispatch in extract_from_file (lines 302-305): .json → extract_from_json (line 332), .html → extract_from_html (line 354).

   • extract_from_json (:332-350): json.dumps(json.loads(raw.decode("utf-8")), indent=2) — same call shape as the GUI path.
   • extract_from_html (:354-371): from api.knowledge import _sanitize_html_content imported at call time, then content, _title = _sanitize_html_content(raw.decode("utf-8", errors="replace")) — calls the real shared sanitiser, not a reimplementation.

   Compare to the GUI path, api/knowledge.py:_extract_file_content (:1075-1109): .csv decoded as text (:1096-1097), .html → _sanitize_html_content (:1101-1103), .json → json.dumps(json.loads(...), indent=2) with the same JSONDecodeError fallback (:1105-1109). Met.

   AC2 — get_supported_extensions() / is_supported_format() report the three formats
   document_extractors.py:470-485 and :487-502 both iterate/query SUPPORTED_FORMATS, which now includes the .csv/.json/.html entries above — no separate list to fall out of sync. Confirmed directly by test_get_supported_extensions_reports_the_three_new_formats and test_is_supported_format_reports_the_three_new_formats in utils/document_extractors_16785_test.py. Met.

   AC3 — a test asserts identical output through both routes for the same bytes
   autobot-backend/utils/document_extractors_16785_test.py imports from api.knowledge import _extract_file_content and from utils.document_extractors import DocumentExtractor, then runs both on the same byte string in each of:

   • test_csv_no_longer_raises_and_matches_upload_path — extractor_text == upload_text == content.decode("utf-8")
   • test_json_reserialised_with_indent_matches_upload_path — asserts equality and the exact indented JSON string
   • test_invalid_json_falls_back_to_raw_text_matching_upload_path — malformed JSON, both routes fall back identically
   • test_html_sanitised_via_the_shared_sanitiser_matches_upload_path — both routes produce the same sanitised text
     Met.

   AC4 — HTML path is not a second sanitiser; genuine negative control
   extract_from_html imports and calls api.knowledge._sanitize_html_content directly (document_extractors.py:363,371) rather than reimplementing tag-stripping. The negative control is test_html_sanitised_via_the_shared_sanitiser_matches_upload_path: input <script>alert(1)</script><p>Hello</p>, asserts "alert(1)" not in extractor_text and "Hello" in extractor_text — the test's own docstring notes this specifically catches "a local reimplementation that merely stripped tags" leaving alert(1) content in. This proves the shared sanitiser's actual script-removal behavior runs through the batch path, not just a superficially similar one. Met.

   All four ACs are met by code on origin/main, not just PR narrative. Closing.
6. mrveiss

   closed this as completedon Sep 19, 2026
7. mrveiss commented on Oct 2, 2026

   

   mrveiss

   on Oct 2, 2026

   OwnerAuthor

   More actions

   AC verification — 4 of 4 delivered, now ticked

   Verified against git show origin/main: at 2b88c1c2e5.

   AC1 — handled with the GUI path's treatment, not decoded as plain text. autobot-backend/utils/document_extractors.py: :97-99 puts .csv in the "text" group with the comment "#16785: .csv joins this group — the GUI upload path (api/knowledge.py …)", while :110-111 give .json and .html their own groups rather than folding them into text. The dispatch docstring at :258-260 states each treatment: .json re-serialised with indent=2, .html sanitised via api.knowledge._sanitize_html_content.

   AC2 — discovery agrees. :101-102: "Listed here so get_supported_extensions() and is_supported_format() — which drive directory discovery — agree with …", and asserted by document_extractors_16785_test.py:95 and :102.

   AC3 — both routes, same bytes, asserted to agree. autobot-backend/utils/document_extractors_16785_test.py:9-11 states the method: "Every test here drives both routes on identical bytes and asserts they agree, since that agreement is …" — the criterion's own reasoning. By test name: :26 csv matches the upload path · :39 json re-serialised matches it · :53 invalid json falls back to raw text and still matches · :65 html sanitised via the shared sanitiser matches · :82 extract_from_file dispatches all three.

   :53 is worth calling out: the agreement is asserted on the malformed input too, which is where two independent decoders would most easily diverge.

   AC4 — not a second sanitiser. :354-371 extract_from_html, docstring: "Reuses api.knowledge._sanitize_html_content rather than a second sanitiser. Imported at call time: this module is reusable by any component, and api.knowledge (a FastAPI router module) is not a …" — then :363 from api.knowledge import _sanitize_html_content and :371 calls it. The criterion verbatim, with the import-time reasoning recorded.

   Part of a sweep over the 93 v0.9.0 closures closed COMPLETED with a checklist and nothing ticked (#17361). Drawn in a seeded random sample.