AI Document Processor

AI-powered document processing pipeline that accepts uploaded files, extracts text, classifies content, and persists structured output. The project is designed as a local-first Python implementation with a deployable AWS architecture built around API Gateway, S3, Step Functions, Lambda, Textract, Comprehend, and DynamoDB.

Why this repo matters

This repo demonstrates how to take a common document-ingestion problem and model it as a reliable event-driven workflow instead of a single monolithic script.

It showcases:

• asynchronous document processing with Step Functions and SQS-style buffering
• OCR and NLP enrichment using Textract-style and Comprehend-style adapters
• separation of ingestion, parsing, classification, and routing concerns
• a local development path that mirrors the cloud design

What is included

• Python package under src/ for ingestion, parsing, classification, and routing
• local CLI for processing sample documents without cloud credentials
• AWS SAM template for the deployable cloud stack
• architecture diagram source in architecture.py
• unit tests under tests/

Architecture

The intended production flow is:

1. A client uploads a document through an API endpoint.
2. The raw file is stored in S3.
3. Step Functions orchestrates parsing and classification work.
4. Processing stages call Textract and Comprehend integrations.
5. Structured results are written to DynamoDB and processed output lands in S3.
6. CloudWatch metrics and alarms surface failures and pipeline health.

Run locally

python3 -m unittest discover -s tests -v
PYTHONPATH=src python3 -m ai_document_processor.cli sample_documents/invoice.txt --output-dir output

Deploy to AWS

sam build
sam deploy --guided

After deployment:

• POST /documents accepts JSON with filename and base64 content
• raw uploads are stored in S3
• the Step Functions workflow processes the document asynchronously
• processed JSON is stored in the processed bucket and DynamoDB

Repository layout

.
|-- sample_documents/   # example local input files
|-- src/                # Python package and processing logic
|-- tests/              # unit tests
|-- template.yaml       # AWS SAM stack
|-- architecture.py     # diagram source
`-- architecture.png    # rendered architecture diagram

What this demonstrates

• serverless workflow design for document pipelines
• practical use of OCR and NLP in an event-driven architecture
• local-first development for a cloud-targeted system
• clean decomposition of processing stages for scale and maintainability

Render the diagram

python architecture.py