An empirical evaluation study and architectural specification analyzing Retrieval-Augmented Generation (RAG) failure modes, prompt engineering constraints, and document retrieval strategies across 160+ enterprise SEC filings (10-Ks, 10-Qs, 8-Ks) and earnings transcripts.
Enterprise competitive intelligence requires precise, fact-grounded synthesis from hundreds of pages of unstructured financial and strategic disclosures. This project evaluates an internal Natural Language Processing (NLP) / Large Language Model (LLM) chatbot designed for Ball Corporation to track strategic customer performance (specifically Molson Coors).
- Baseline Retrieval Accuracy: The system achieved a 63% baseline factual accuracy rate across complex multi-document financial queries.
- Order-of-Operations Bias: Uncovered that the LLM processes prompt structural constraints sequentially rather than holistically (e.g., placing format constraints at the beginning of the prompt resolved paragraph-count truncation errors).
- Retrieval Bottlenecks Identified: Uncovered primary RAG degradation points including sequential retrieval bias, context-window saturation (capping citations at 3 documents), and cross-entity pollution.
| Failure Mode | Root Cause in RAG Pipeline | Observed Impact |
|---|---|---|
| Sequential Retrieval Bias | Naive top-$k$ similarity search without temporal re-ranking | The model cited the first available document rather than the most recent/authoritative filing. |
| Citation Truncation | Rigid context chunk retrieval limits | LLM capped retrieval at 3 documents, failing to aggregate data across multi-year trends. |
| Cross-Entity Confusion | Lack of named-entity recognition (NER) pre-filtering | Trigger words (e.g., "marketing campaign") retrieved unrelated competitor filings (e.g., Coca-Cola). |
| Prompt Instruction Decay | End-of-prompt constraint placement | Format instructions placed at the end of long prompts were ignored; front-loading constraints restored compliance. |
To resolve the 37% error margin, the report outlined four targeted technical enhancements:
- Hierarchical Document Indexing: Implement a two-tier retrieval architecture (searching document-level metadata/titles before content chunks) to prevent irrelevant section scanning.
- Temporal & Metadata Re-Ranking: Apply recency-weighted re-ranking algorithms so newer quarterly earnings and 8-K filings override older disclosures.
- Chain-of-Thought (CoT) Verification: Enforce step-by-step verification prompts ("think twice", self-correction queries) to reduce hallucinated financial ratios.
- Automated Evaluation Framework: Transition from manual ad-hoc testing to a systematic evaluation benchmark scoring factual recall, faithfulness, and hallucination rate.
reports/: Complete technical whitepaper and business case analysis.evaluation_data/: Empirical prompt-response benchmark logs tracking query accuracy, model responses, and error categories.