<maintenance> maintaining the experimental branch via main

.gitignore

@@ -50,7 +50,6 @@ var/
 sdist/
 develop-eggs/
 .installed.cfg
-lib/
 lib64/
 *.egg
 
@@ -80,3 +79,15 @@ next-env.d.ts
 .env.development.local
 .env.test.local
 .env.production.local
+
+# MCP Server specific
+fast-markdown-mcp/venv/
+fast-markdown-mcp/__pycache__/
+fast-markdown-mcp/**/__pycache__/
+fast-markdown-mcp/build/
+fast-markdown-mcp/dist/
+fast-markdown-mcp/*.egg-info/
+
+# Service logs
+logs/
+*.log

README.md

@@ -15,15 +15,15 @@ The idea of DevDocs is to ensure that software engineers and (LLM) software devs
 ## Roadmap:
 - [X] ~~Build a Minimum Viable Product with accurate functionality~~
 - [ ] Handle Complex websites documentation like AWS, MS, langchain :D 
-- [ ] Adding MCP servers options to choose.
+- [X] ~~Adding MCP servers options to choose.~~
 - [ ] Turnkey Vector Database so all chunking, embedding is done behind the scenes while you sip your joe :) 
 - [ ] Agents which will be pros in particular documentation and can code, architect or reason for you with the accurate information as ground truth.
 - [ ] Option to switch to LLM based crawling for specific usecase.
 - [ ] UI Improvements, cuz we flashy. 
 
 
+![image](https://github.com/user-attachments/assets/8bdc3dfe-1fb9-4ace-8259-e6155f44ebcd)
 
-![image](https://github.com/user-attachments/assets/40f9e0b0-b662-46bf-821a-4dab23c54649)
 
 ## ✨ Features
 
@@ -33,75 +33,31 @@ The idea of DevDocs is to ensure that software engineers and (LLM) software devs
 - 🎯 **Precision Extraction**: Focuses on meaningful content while filtering out noise
 - 🚄 **Real-time Progress**: Live updates on crawling progress and statistics
 - 💫 **Modern UI**: Sleek, responsive interface with real-time feedback
-
-## 🏗️ Architecture
-
-### Frontend (Next.js + TypeScript)
-- Modern React components with TypeScript
-- Real-time state management with React hooks
-- Tailwind CSS for styling
-- Shadcn UI components for consistent design
-
-### Backend (FastAPI + Python)
-- FastAPI for high-performance async API
-- Crawl4AI for intelligent web crawling
-- Advanced error handling and recovery
-- Session management for reliable crawling
+- 🔥 **Inbuilt MCP Server**: No need to copy paste into your MCP server, DevDocs already has an inbuild MCP server, already connect to your claude desktop app upon installation(restart needed) and gives you the commands you need to add to your cline MCP server configs. How cool is that? 
 
 ## 🚀 Getting Started
 
-### Prerequisites
-- Node.js 18+
-- Python 3.12+
-- npm or yarn
-- pip
-
-### Installation
-
-1. Clone the repository:
 ```bash
-git clone https://github.com/cyberagiinc/DevDocs.git
-cd DevDocs
-```
-
-2. Install frontend dependencies:
-```bash
-npm install
-# or
-yarn install
-```
+# 1. Clone and install everything
+git clone https://github.com/cyberagiinc/DevDocs.git && cd DevDocs && ./fast-markdown-mcp/setup.sh
 
-3. Install backend dependencies:
-```bash
-cd backend
-pip install -r requirements.txt
-```
-
-### Running the Application
-
-1. Start the backend server:
-```bash
-cd backend
-uvicorn app.main:app --host 0.0.0.0 --port 24125 --reload
-```
-
-2. Start the frontend development server:
-```bash
-# In another terminal
-npm run dev
-# or
-yarn dev
+# 2. Start all services
+./start.sh
 ```
 
-3. Open [http://localhost:3000](http://localhost:3000) in your browser
+That's it! The system will:
+- Install all dependencies (npm, Python backend, MCP server)
+- Configure Claude Desktop integration
+- Start all services automatically
+- Open the application in your browser (http://localhost:3001)
 
 ## 💡 How to Use DevDocs: Super Difficult :) 
 
 1. Enter a documentation URL (e.g., [https://docs.crawl4ai.com](https://docs.crewai.com/))
 2. Click "Discover" to find all related pages
 3. Review the discovered pages in the list
 4. Click "Crawl All Pages" to extract content. Go get some coffee, because it takes a while. 
-5. Download or copy the generated markdown
+5. Download the generated markdown/json or use it with an inbuilt MCP server with Cline/Claude
 
 ## 🤝 Contributing
 
@@ -119,4 +75,4 @@ Apache-2.0 license
 
 [![Star History Chart](https://api.star-history.com/svg?repos=cyberagiinc/DevDocs&type=Timeline)](https://star-history.com/#cyberagiinc/DevDocs&Timeline)
 
-Built with ❤️ by CyberAGI Inc. | Report Issues like you would do for any Github repo, if you know how to fix it, provide a fix using the Contribution method. 
+Built with ❤️ by CyberAGI Inc. & Shubham Khichi

app/api/storage/download/route.ts

@@ -0,0 +1,74 @@
+import { NextResponse } from 'next/server'
+import fs from 'fs/promises'
+import path from 'path'
+
+export async function GET(request: Request) {
+  try {
+    const { searchParams } = new URL(request.url)
+    const filePath = searchParams.get('path')
+
+    if (!filePath) {
+      return NextResponse.json(
+        { success: false, error: 'No file path provided' },
+        { status: 400 }
+      )
+    }
+
+    // Security check to ensure the path is within the storage directory
+    const storagePath = path.join(process.cwd(), 'storage/markdown')
+    const normalizedPath = path.normalize(filePath)
+    if (!normalizedPath.startsWith(storagePath)) {
+      return NextResponse.json(
+        { success: false, error: 'Invalid file path' },
+        { status: 403 }
+      )
+    }
+
+    // Check if file exists
+    try {
+      await fs.access(normalizedPath)
+    } catch {
+      return NextResponse.json(
+        { success: false, error: 'File not found' },
+        { status: 404 }
+      )
+    }
+
+    // Read the file
+    const content = await fs.readFile(normalizedPath, 'utf-8')
+
+    // If it's a JSON file, verify it's valid JSON
+    if (path.extname(filePath) === '.json') {
+      try {
+        JSON.parse(content)
+      } catch {
+        return NextResponse.json(
+          { success: false, error: 'Invalid JSON file' },
+          { status: 500 }
+        )
+      }
+    }
+
+    // Determine content type based on file extension
+    const contentType = path.extname(filePath) === '.json' 
+      ? 'application/json' 
+      : 'text/markdown'
+
+    // Create response with appropriate headers for download
+    return new NextResponse(content, {
+      headers: {
+        'Content-Type': contentType,
+        'Content-Disposition': `attachment; filename="${path.basename(filePath)}"`,
+      },
+    })
+  } catch (error) {
+    console.error('Error downloading file:', error)
+    return NextResponse.json(
+      { 
+        success: false, 
+        error: error instanceof Error ? error.message : 'Failed to download file' 
+      },
+      { status: 500 }
+    )
+  }
+}