arxiv2code

Paste an arXiv paper. Get a real implementation kit.

🔗 Live demo: arxiv2code.vercel.app

arxiv2code is an AI-powered tool that turns any arXiv machine learning paper into a structured implementation kit — an engineering blueprint, extracted hyperparameter config, starter PyTorch code, and a reproducibility checklist. It's built for engineers who need to implement a paper, not just read about it.

The problem

Only around 20% of top ML conference papers ship official code. Engineers routinely spend days reverse-engineering a method from dense academic prose — figuring out architecture details, missing hyperparameters, and ambiguous implementation choices by hand. Tools like Elicit and SciSpace help researchers find and summarize papers. Nothing helps engineers implement them. arxiv2code closes that gap.

What it generates

For any arXiv URL, arxiv2code produces:

• Implementation blueprint — the method section turned into concrete, numbered engineering tasks with paper section citations, not academic summaries
• Config extraction — every hyperparameter, dataset spec, and hardware requirement mentioned in the paper, pulled into one structured JSON object
• Starter code skeleton — Python/PyTorch class and function stubs referencing the exact paper sections they implement
• Reproducibility checklist — an honest list of what the paper doesn't specify, so you know exactly where you'll have to make your own calls

How it works

1. Ingestion — fetches arXiv metadata, downloads the PDF, extracts text with PyMuPDF, and splits it into sections using heading-aware regex.
2. Generation — three independently-tuned LLM prompts (blueprint, config, code) run against the relevant sections, with a multi-provider fallback chain (Groq → NVIDIA → Gemini) so a single provider's rate limit doesn't take the product down.
3. Reproducibility check — a purely programmatic cross-reference between the extracted config and the generated blueprint's own ambiguity flags — no extra LLM call needed.
4. Retrieval layer — sections are chunked and embedded (sentence-transformers + FAISS) for semantic search across ingested papers.

The backend is hardened for real traffic: structured per-request logging, retry-with-backoff on every external dependency, rate limiting, and load-tested concurrency fixes (async LLM clients, connection pooling, race-condition-safe DB writes).

Tech stack

Backend: FastAPI · SQLAlchemy (SQLite locally, Postgres in production) · PyMuPDF · httpx · Groq / NVIDIA NIM / Gemini APIs · slowapi rate limiting · structured logging

Frontend: Next.js · TypeScript · Tailwind CSS · shadcn/ui · Framer Motion

Evaluation: a 10-paper eval harness with LLM-as-judge scoring across four dimensions — blueprint completeness, config accuracy, code skeleton relevance, and ambiguity detection — with automated failure-pattern diagnosis. See eval_report.md for real results.

Project structure

arxiv2code/
├── main.py              # FastAPI app, routes, middleware, error handling
├── ingestion.py          # arXiv metadata + PDF fetch orchestration
├── arxiv_client.py       # arXiv API client, PDF text extraction, section splitting
├── chunking.py           # section-aware chunking for embeddings
├── embeddings.py         # sentence-transformers embedding generation
├── build_index.py        # FAISS index builder
├── search.py              # standalone retrieval test script
├── llm_client.py         # multi-provider LLM client with retry + fallback chain
├── judge_client.py       # LLM-as-judge client for evals
├── implement.py           # implementation kit generation pipeline
├── prompts.py             # all LLM system prompts
├── models.py / database.py # SQLAlchemy models and DB config
├── logging_utils.py      # structured, request-ID-correlated logging
├── locustfile.py           # load test scenarios
├── eval_set.py / run_eval.py / run_judge.py / eval_report.py  # Day 5 eval harness
└── frontend/               # Next.js app (see below)

Running locally

Backend:

cd arxiv2code
python -m venv venv
source venv/bin/activate  # or venv\Scripts\activate on Windows
pip install -r requirements.txt
cp .env.example .env  # fill in your API keys
uvicorn main:app --reload

Frontend:

cd frontend
npm install
npm run dev

Set NEXT_PUBLIC_API_URL in frontend/.env.local to point at your backend.

Deployment

• Backend is deployed on Render via Docker
• Frontend is deployed on Vercel
• Database: SQLite for local dev; Postgres in production for persistence across redeploys

About

Built by Muhammad Farhan — a developer who just loves trying new things.

• Contact
• LinkedIn
• Datraxa