Skip to content

Add zip/archive recursive ingestion for KB document uploads #16774

Description

@mrveiss

Problem

AutoBot has no path anywhere that opens a .zip archive and ingests its members as documents. Confirmed absent: grep -rli "zipfile" across autobot-backend/media/ and autobot-backend/knowledge/ returns no hits; ALLOWED_EXTENSIONS in autobot-backend/api/knowledge.py:126 has no .zip entry; DocumentExtractor.SUPPORTED_FORMATS (autobot-backend/utils/document_extractors.py) has no zip entry either. A user with a folder of documents must upload each file individually.

Location

  • autobot-backend/api/knowledge.py:126 (ALLOWED_EXTENSIONS)
  • autobot-backend/utils/document_extractors.py (DocumentExtractor.SUPPORTED_FORMATS)
  • New extractor, likely autobot-backend/media/document/archive.py, following the existing module split in media/document/ (extraction.py, provenance.py, ocr.py)

Impact

Bulk KB uploads (a folder of research PDFs, a project's docs) require N individual uploads today instead of one zip upload — a real workflow friction for any user onboarding an existing document set.

Acceptance Criteria

  • A .zip upload is unpacked and each supported member is routed through the existing per-format extractors (extract_document, DocumentParser) — no new parsing logic duplicated
  • Hard caps enforced and configurable via the existing env-var-backed-constant pattern (see min_text_page_ratio()/max_table_pages() in media/document/extraction.py for the pattern to follow): member count, nesting depth, and total decompressed size — this is a decompression-bomb vector and ships with the caps, not as a follow-up
  • Each member's extraction respects the existing per-file extraction_timeout() (media/document/ocr.py) individually, not one timeout for the whole archive
  • An unsupported or corrupt member inside the archive is skipped and reported, not aborting the whole batch (matches existing per-file failure handling in DocumentExtractor.extract_from_directory)
  • Test coverage includes a zip bomb / oversized-member fixture asserting the caps actually trip

Discovered During

Research pass — docs/research/document-to-markdown-conversion-pipeline.md, "What We Can Adopt" #2

Activity

  1. mrveiss commented on Sep 16, 2026

    @mrveiss
    OwnerAuthor

    Ordering call from the coordinator: recorded blocked_by #16773. Recursive archive ingestion widens what a single upload can put into the KB, so the content-verification work should land first — otherwise the new path inherits extension-based trust for every member of the archive, which is the exact gap #16773 closes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions