🏗️ Master Core Infrastructure Restructure

[r3d91ll]

🏗️ Master Core Infrastructure Restructure

Executive Summary

Complete restructuring of the core/ directory to eliminate redundancy, establish clear separation of concerns, and create a sustainable architecture for future development. This is a 21-day sprint to restructure the core infrastructure while maintaining all functionality.

🎯 Objectives

1. Eliminate Redundancy: Remove core/framework/ layer (everything in core IS the framework)
2. Clear Separation: Establish distinct modules with single responsibilities
3. Consistent Naming: All files prefixed with parent directory name
4. Centralized Configuration: Single source of truth for all settings
5. Validation Through Rebuild: Use database rebuild as integration test

📋 Current State

Baseline Metrics (2025-01-13)

• Papers Processed: 2,823,744
• Embeddings Generated: 2,823,744 (100% coverage)
• Processing Speed: 48.3 papers/second
• Total Time: 399.1 minutes for 1.16M papers
• Workers: 2 GPU workers, batch size 80
• Model: jinaai/jina-embeddings-v4 (2048 dimensions)

Issues Identified

• core/framework/ is redundant - everything in core IS the framework
• core/graph/ contains 12+ duplicate edge builders
• Configuration scattered across multiple systems
• Storage functionality split between multiple locations
• Monitoring inconsistently implemented

🏛️ New Architecture

core/
├── embedders/           # All embedding models (Jina, VLLM, etc.)
├── extractors/          # Document extraction (Docling, PDFPlumber, etc.)
├── processors/          # Processing pipelines
│   ├── text/           # Text processing (chunking, cleaning)
│   ├── structured/     # Tables, equations, images
│   └── code/           # Tree-sitter, symbol extraction
├── database/           # Database interfaces
│   ├── arango/        # ArangoDB client and operations
│   └── postgres/      # PostgreSQL operations
├── workflows/          # Orchestration and pipelines
│   ├── storage/       # Storage backends (S3, local, etc.)
│   └── state/         # State management
├── monitoring/         # Metrics and observability
├── config/            # Centralized configuration
├── utils/             # True utilities only
└── mcp_server/        # MCP interface (unchanged)

📊 Sub-Issues (To Be Created)

Phase 1: Foundation (Days 1-3)

• 🏗️ Master Core Infrastructure Restructure #35 - Move embedders from framework/ to embedders/
• feat: Phase 1 - Core Restructure: Embedders and Extractors Migration #36 - Move extractors from framework/ to extractors/
• 📝 Add docstrings to feature/core-restructure-phase1 #37 - Reorganize processors/ with subdirectories

Phase 2: Database & Workflows (Days 4-7)

• feat: Phase 2 - Database and Workflows Consolidation #38 - Consolidate database interfaces
• feat: Phase 3 - Configuration and Monitoring Consolidation (Issue #35) #39 - Create workflows/ with storage integration
• feat: Phase 4 - Integration and Testing (Issue #35) #40 - Implement centralized configuration

Phase 3: Monitoring & Utils (Days 8-10)

• Feature/core restructure phase5 e2e #41 - Consolidate monitoring from all modules
• feat: Implement dual embedders with mandatory late chunking #42 - Clean utils/ to true utilities only

Phase 4: Integration & Testing (Days 11-15)

• Feature/dual embedders late chunking 42 #43 - Update all imports across codebase
• Critical: Fix production issues in workflow_arxiv_sorted_simple.py #44 - Run integration tests
• Fix Jina Embedder Issues and Config Integration #45 - Rebuild database for validation

Phase 5: Documentation & Cleanup (Days 16-18)

• Remove redundant SentenceTransformersEmbedder and standardize on JinaV4 #46 - Update all documentation
• fix: Issues #44, #45, #46 - Critical embedder and workflow fixes #47 - Archive old code to Acheron/
• 📝 Add docstrings to feature/issue-46-remove-sentence-transformers #48 - Update CLAUDE.md

Phase 6: Final Validation (Days 19-21)

• Feature: Implement gRPC Database Service Layer #49 - Compare metrics with baseline
• Feature: Go gRPC Memory Service for Local AI Models #50 - Performance benchmarking
• feat: Implement HTTP/2 optimized ArangoDB client with neural process isolation #51 - Sign-off and merge

✅ Success Criteria

1. Functional Validation

   • Database rebuild produces identical results (2,823,744 papers)
   • Maintain or exceed 48.3 papers/second throughput
   • All tests pass

2. Architectural Goals

   • No redundant layers (framework/ eliminated)
   • Clear module boundaries
   • Consistent naming throughout
   • Single configuration source

3. Code Quality

   • All imports updated and working
   • Documentation complete
   • Old code archived with timestamps

🚫 Out of Scope

• Graph Module Refactor: Deferred until after validation (too complex for this sprint)
• New Features: Focus only on restructuring existing functionality
• Algorithm Changes: Maintain exact same processing logic

📁 Reference Documents

• Master PRD: /docs/prd/core_restructure/master-core-restructure-prd.md
• Baseline Snapshot: /docs/baseline/baseline-2025-01-13.md
• Database Snapshot: /tools/arxiv/utils/baseline-snapshot.json

🔄 Related Issues

• Supersedes feat: Implement Centralized Configuration System #34 (Configuration System PRD)
• References baseline commit: cc85ff2

📅 Timeline

• Start Date: January 13, 2025
• End Date: February 3, 2025
• Duration: 21 days

🎯 Definition of Done

• All sub-issues completed
• Database rebuild matches baseline (2,823,744 papers)
• Performance meets or exceeds baseline (48.3 papers/sec)
• All tests passing
• Documentation updated
• Old code archived to Acheron/
• CLAUDE.md updated
• PR approved and merged to main

---

Note: Graph module refactor (#52) will be created as a separate epic AFTER this restructure is validated.

[r3d91ll]

Closing as Substantially Complete

✅ Completed:

• Removed core/framework/ - Already gone (everything in core IS the framework)
• Eliminated redundant embedders - Consolidated to JinaV4Embedder
• Centralized configuration - Created /core/config/ with YAML configs
• Archeon protocol - Established for preserving deprecated code
• Clear module separation - Extractors, embedders, workflows all properly organized

📌 Intentionally Preserved:

• core/graph/ - Keeping duplicate builders for next phase (graph development)
• This is our next major development phase, so these will be addressed then

Current Structure is Clean:

core/
├── config/          # Centralized configuration ✅
├── database/        # Storage layer ✅
├── embedders/       # Consolidated embedders ✅
├── extractors/      # Document extractors ✅
├── graph/           # Graph builders (next phase)
├── monitoring/      # Monitoring utilities ✅
├── processors/      # Text processors ✅
├── utils/           # Shared utilities ✅
└── workflows/       # Processing workflows ✅

The major restructuring goals have been achieved. Any remaining improvements can be handled incrementally or as part of the graph development phase.

Closing this issue as the core restructure is complete.