Fix/10 bug fixes excel viewer - #10
Conversation
- Rich Excel evidence panel with DIGITAL LEDGER badge, balance stats, download button - Non-PDF/non-Excel files get clean fallback with download link - Cleaned database of 67 old junk documents (fresh start) - Added node_modules, *.db, storage, backup dirs to .gitignore
- Account 3 fraud: detect header vs calculated balance discrepancy - New metadata_discrepancy field on NormalizedStatement - Header closing (-61M) vs calculated (~61K) -> FRAUD SIGNAL - Confidence slammed to 12% max (red badge) for metadata failures - Injected as validation error so it shows in Review UI - NO LONGER silently overrides the fraudulent closing balance - Account 4 time-travel: date-sequence warnings consolidated - Deduplicated: one warning with violation count instead of per-row - Severity escalated to 'warning' (1-4 violations) or 'critical' (5+) - Penalties: -3% per warning, -10% per critical -> yellow badge - Added date_sequence and metadata_integrity to anomaly type_map
…act hooks crash, add dynamic Lie Detector panel
- Fix metadata_discrepancy not persisting to SQLite extracted_fields JSON
- Add structured fields (header_closing, calculated_closing, discrepancy, ratio) to anomaly details
- Replace float('inf') with safe integer 999999999 for JSON serialization
- Fix React hooks ordering violation - move all hooks before early returns
- Add dynamic Lie Detector integrity panel with 3-priority fallback:
1. Backend-computed metadata_discrepancy from normalizer
2. Anomaly details with structured integrity data
3. Local closing_balance vs last transaction comparison
- Clean hardcoded Account references from normalizer
- Remove debug instrumentation from ingestion pipeline
There was a problem hiding this comment.
Pull request overview
This PR improves the Excel/CSV document review experience by adding a dedicated “digital ledger” preview state in the web UI and introducing backend-side metadata integrity checks (header closing vs calculated closing) that surface as anomalies/errors and can drive confidence penalties.
Changes:
- Web UI: add an Excel/CSV preview panel with integrity/fraud status and a download action; adjust early returns to comply with React hook ordering.
- Backend: add a date-sequence validation warning with an aggregated violation count and severity.
- Backend: compute and propagate
metadata_discrepancyfrom the Excel normalizer through ingestion, penalize confidence, and emit a validation error/anomaly for UI visibility.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
webapp/src/pages/DocumentReviewPage.tsx |
Adds Excel/CSV preview state and a dynamic integrity check; moves early returns after hooks. |
backend/app/services/validation.py |
Aggregates out-of-order transaction date checks into a single warning with count/severity. |
backend/app/services/excel_normalizer.py |
Computes and records metadata_discrepancy + anomaly for header/row closing mismatch fraud signal. |
backend/app/api/ingestion.py |
Ensures extracted_fields is mutable, persists new anomaly mappings, passes discrepancy to frontend, and applies confidence/error injection for fraud signals. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| classification = analysis.get("classification", {}) | ||
| extracted_fields = analysis.get("extracted_fields", {}) | ||
| # Ensure extracted_fields is a mutable plain dict (some backends return special objects) | ||
| extracted_fields = dict(analysis.get("extracted_fields", {})) |
There was a problem hiding this comment.
extracted_fields = dict(analysis.get("extracted_fields", {})) will raise TypeError if Backboard returns an explicit null for extracted_fields (the key exists but value is None). Use a null-safe fallback before copying (e.g., analysis_extracted = analysis.get("extracted_fields") or {} then extracted_fields = dict(analysis_extracted)).
| extracted_fields = dict(analysis.get("extracted_fields", {})) | |
| analysis_extracted = analysis.get("extracted_fields") or {} | |
| extracted_fields = dict(analysis_extracted) |
| last_balance = None | ||
| # Step 5: Metadata Integrity Check | ||
| # Compare the header/summary closing balance against the actual last | ||
| # transaction balance. A massive discrepancy (>1.0 AND >50x) means the |
There was a problem hiding this comment.
The comment says the integrity threshold is ">1.0 AND >50x", but the code flags when the header is >5x the calculated closing (abs(header_closing) > abs(calculated_closing) * 5). Please align the comment and implementation (either update the comment to 5x or adjust the multiplier) to avoid future confusion about fraud thresholds.
| # transaction balance. A massive discrepancy (>1.0 AND >50x) means the | |
| # transaction balance. A massive discrepancy (>1.0 AND >5x) means the |
No description provided.