Autonomous Enterprise Intelligence Agent

A GraphRAG system that cross-references live GitHub PR diffs against OWASP ASVS v4.0 and NIST 800-53 Rev. 5 compliance standards in real time — preventing security misconfigurations before they reach production.

Live demo: https://rag-vercel-capstone.vercel.app

---

What it does

When a pull request is labeled security in GitHub, the system:

1. Receives a webhook, verifies the HMAC signature, and checks idempotency
2. Fetches the PR diff and chunks it into overlapping token windows
3. Scans the diff for secrets and masks PII before any storage
4. Embeds code chunks and runs hybrid vector + BM25 search against 573 pre-indexed OWASP/NIST regulation chunks
5. Augments retrieval with a Neo4j graph traversal (e.g. resolving superseded controls, following GOVERNS edges from PR → File → Component → Control)
6. Re-ranks the top 20 candidates down to 5 using GPT-4o-mini
7. Synthesizes a cited compliance answer using Claude 3.5 Sonnet
8. Routes low-confidence results (< 0.55) to a Slack HITL approval card
9. Logs every step to Supabase and emits OpenTelemetry spans to Grafana Tempo

---

Architecture

GitHub PR (webhook)
        │
        ▼
   n8n Orchestrator
   ├── HMAC verify
   ├── Idempotency check (Supabase audit_log)
   ├── Fetch diff → webhook_processor (FastAPI :8001)
   │     ├── Secret scan + PII mask
   │     ├── Chunk + embed (text-embedding-3-small)
   │     └── Upsert → Supabase code_embeddings
   └── Query pipeline (FastAPI :8002)
         ├── Hybrid search  (pgvector cosine + BM25 → RRF)
         ├── Graph augment  (Neo4j Cypher — superseded controls, GOVERNS edges)
         ├── Rerank         (GPT-4o-mini, top-20 → top-5)
         ├── Safety check   (adversarial pattern detection)
         ├── Synthesize     (Claude 3.5 Sonnet, paragraph citations)
         ├── HITL trigger   (Slack card if confidence < 0.55)
         ├── Audit log      (Supabase, async)
         └── OTel traces    (10 spans → Grafana Tempo)

---

Tech Stack

Component	Technology
Orchestrator	n8n (webhook-driven, ~100 executions/mo)
Ingestion + Query API	FastAPI / Python
Vector DB	Supabase pgvector (IVFFlat index, 573 regulation chunks)
Graph DB	Neo4j AuraDB (Framework → ControlFamily → Control → PR → File → Component)
Hybrid search	pgvector cosine + BM25, merged via Reciprocal Rank Fusion
Re-ranker	GPT-4o-mini (pairwise scoring, top-20 → top-5)
Synthesis	Claude 3.5 Sonnet (paragraph-level citations)
Safety layer	Regex + pattern detection for adversarial inputs
HITL	Slack approval cards with 2h / 4h escalation
Observability	OpenTelemetry (10 spans), Grafana Tempo
Frontend	Next.js 14, deployed on Vercel
Regulations	OWASP ASVS v4.0 (CC BY-SA 4.0) + NIST 800-53 Rev. 5 (public domain)

---

Evaluation Results

30-pair eval set — 25 standard queries + 5 adversarial:

Metric	Result	Target	Status
Groundedness	96%	≥ 90%	PASS
Exact clause match	88%	≥ 80%	PASS
Avg factuality	88.6%	≥ 85%	PASS
HITL rate	20%	≥ 10%	PASS
Adversarial blocked	5 / 5	5 / 5	PASS
p95 latency	24.3s	≤ 20s	documented*
Cost per query	$0.024	≤ $0.01	documented*

*Latency and cost are architectural trade-offs of a 3-model pipeline (embed → rerank → synthesize). Documented in enterprise-intelligence-agent/ingestion/eval_known_failures.md.

---

Integration Tests (5/5 pass)

cd enterprise-intelligence-agent/ingestion
# Requires ports 8001 (webhook_processor) and 8002 (query_pipeline) running
python test_integration.py

Test	What it verifies
Compliance mapping	PR query returns NIST AC-2 with canonical ID + section citation
Superseded policy resolution	SUPERSEDES graph traversal returns current control, not deprecated one
HITL trigger	Low re-ranker scores fire correct Slack payload
Safety block	Adversarial PR diff is rejected and logged to DLQ
Webhook + HMAC	Real webhook fires pipeline within 30s with HMAC verified

---

Repository Structure

├── enterprise-intelligence-agent/
│   ├── ingestion/              # FastAPI services and pipeline scripts
│   │   ├── webhook_processor.py    # Ingestion: HMAC, PII masking, chunking, embedding
│   │   ├── query_pipeline.py       # Query API: search, rerank, synthesize, audit
│   │   ├── hybrid_search.py        # pgvector + BM25 + RRF
│   │   ├── neo4j_search.py         # Cypher graph augmentation
│   │   ├── reranker.py             # GPT-4o-mini pairwise reranker
│   │   ├── synthesizer.py          # Claude 3.5 Sonnet synthesis
│   │   ├── safety.py               # Adversarial input detection
│   │   ├── hitl.py                 # Slack HITL card dispatch
│   │   ├── audit.py / auditor.py   # Async audit logging to Supabase
│   │   ├── ingest_regulations.py   # One-time PDF ingestion (OWASP + NIST)
│   │   ├── seed_graph.py           # Neo4j graph seeding
│   │   ├── eval_runner.py          # 30-pair evaluation harness
│   │   ├── eval_results.json       # Final eval metrics
│   │   ├── eval_known_failures.md  # Documented architectural trade-offs
│   │   ├── test_integration.py     # 5 integration tests
│   │   ├── test_dlq.py             # DLQ integration tests
│   │   └── requirements.txt
│   └── n8n/
│       ├── github_pr_workflow.json # Import into n8n: PR ingestion pipeline
│       └── query_workflow.json     # Import into n8n: query routing workflow
├── rag-vercel-capstone/        # Next.js frontend (4 pages)
│   ├── app/
│   │   ├── page.tsx                # Dashboard — query interface + confidence display
│   │   ├── traces/[trace_id]/      # Trace detail — OTel span viewer
│   │   ├── admin/                  # Admin — DLQ management
│   │   └── attributions/           # Attributions — OWASP/NIST license compliance
│   └── components/
├── supabase_schema.sql         # Full Supabase schema (5 tables)
├── capstone_proposal/          # Original project proposal
└── .env.example                # Environment variable template

---

Setup

1. Clone and install

git clone https://github.com/your-github-username/dataexpert-ai-capstone
cd dataexpert-ai-capstone
python -m venv venv && source venv/bin/activate
pip install -r enterprise-intelligence-agent/ingestion/requirements.txt

2. Configure environment

cp .env.example .env
# Fill in your Supabase, Neo4j, OpenAI, Anthropic, GitHub, and Grafana credentials

3. Run the services

# Terminal 1 — Ingestion API
uvicorn webhook_processor:app --host 0.0.0.0 --port 8001 --reload

# Terminal 2 — Query API
uvicorn query_pipeline:app --host 0.0.0.0 --port 8002 --reload

# Terminal 3 — expose query pipeline for Vercel frontend (ngrok or similar)
ngrok http 8002

4. Import n8n workflows

In your n8n instance: Settings → Import workflow → import both JSON files from enterprise-intelligence-agent/n8n/.

5. Apply Supabase schema

Run supabase_schema.sql in the Supabase SQL editor.

---

Confidence Scoring

confidence = (reranker_mean × 0.5) + (citation_density × 0.3) + (retrieval_overlap × 0.2)

Band	Range	Action
Verified	> 0.70	Return answer directly
Warn	0.55 – 0.70	Return with caution flag
HITL	< 0.55	Route to Slack for human review

---

Regulatory Attribution

• OWASP ASVS v4.0 — licensed under CC BY-SA 4.0. No modifications made; used for compliance cross-referencing only.
• NIST 800-53 Rev. 5 — U.S. Government work, public domain. See NIST disclaimer.