AI-Driven Adaptive Diagnostic Engine

A professional-grade 1-Dimension Adaptive Testing prototype designed for GRE preparation. This system dynamically adjusts question difficulty in real-time based on student performance using Item Response Theory (IRT).

🚀 Features

• Adaptive Engine: Implements a 1PL IRT model (Rasch Model) to estimate student ability and select optimal questions.
• Dynamic Question Selection: Maximizes Fisher Information to choose the most informative next question from a seeded database.
• Modern UI: A clean, professional interface built with vanilla HTML/CSS/JS (no frameworks) focusing on clarity and accessibility.
• AI-Powered Insights: Integrates with OpenRouter (LLM) to generate personalized 3-step study plans based on test performance.
• Real-time Analytics: Visualizes ability score progression and topic-wise performance.
• User Authentication: Clerk-powered authentication allows users to sign in and track their test history across sessions.

---

🛠️ Tech Stack

• Backend: Python 3.10+, FastAPI
• Database: MongoDB Atlas
• AI Integration: OpenRouter API (using Llama 3.3 70B)
• Authentication: Clerk (JWT-based)
• Frontend: Vanilla JavaScript, CSS Grid/Flexbox, Canvas API
• Algorithm: Item Response Theory (IRT) - 1-Parameter Logistic Model

---

🏃 Instructions to Run

1. Prerequisites

• Python 3.10 or higher installed
• A MongoDB Atlas connection string
• An OpenRouter API key (for study plan generation)
• A Clerk account with API keys (for authentication)

2. Setup Environment

Clone the repository and create a .env file from the provided template:

cp .env.example .env
# Then edit .env with your actual API keys

Required environment variables (see .env.example):

OPENROUTER_API_KEY=your_api_key_here
MONGODB_URI=mongodb://localhost:27017
DATABASE_NAME=adaptive_diagnostic
LLM_MODEL=meta-llama/llama-3.3-70b-instruct:free
ALLOWED_ORIGINS=http://localhost:8000

# Clerk Authentication (get keys from dashboard.clerk.com)
CLERK_PUBLISHABLE_KEY=pk_test_your_key_here
CLERK_SECRET_KEY=sk_test_your_key_here

3. Install Dependencies

cd backend
pip install -r requirements.txt

4. Seed the Database

Populate MongoDB with the GRE-style question bank (25+ questions):

python -m app.seed

5. Start the Server

python -m uvicorn app.main:app --reload

The application will be available at http://127.0.0.1:8000.

---

🐳 Docker

Run the entire stack (app + MongoDB) with Docker Compose:

cp .env.example .env
# Edit .env with your API keys

docker compose up --build

This starts MongoDB on port 27017 and the app on port 8000. Seed the database:

docker compose exec app python -m app.seed

---

🧪 Testing

The project includes a comprehensive test suite (74 tests) covering IRT math, schema validation, API endpoints, and resource integrity.

cd backend
pip install -r requirements.txt  # includes pytest + pytest-asyncio
python -m pytest tests/ -v

Test breakdown:

• test_irt_engine.py (18 tests) — Probability model, ability updates, Fisher information, target difficulty selection
• test_schemas.py (21 tests) — Pydantic validation, XSS/SQL injection sanitization, boundary values, ResourceLink model, StudyPlan bounds
• test_api.py (23 tests) — All 10 endpoints with mocked MongoDB, error handling (404/400/500), auth edge cases, study plan success/failure, question stats, user history, user profile
• test_resource_map.py (6 tests) — Topic coverage, required fields, HTTPS enforcement, fallback to General resources
• conftest.py — Shared fixtures for sample questions and sessions

---

🔐 Authentication

The application uses Clerk for user authentication:

• Sign Up / Sign In: Users can create accounts or sign in via the welcome screen
• Guest Mode: Users can also take tests without signing in (anonymous sessions)
• Test History: Authenticated users can view their past test sessions and track progress
• Persistent Sessions: User data is linked via clerk_user_id in MongoDB

How It Works

1. ClerkJS loads on the frontend and manages the sign-in/sign-up UI
2. When authenticated, the frontend fetches a JWT token from Clerk
3. API calls include the token in the Authorization: Bearer header
4. Backend verifies the JWT against Clerk's JWKS endpoint
5. User ID from the token is stored with each test session

---

🧠 Adaptive Algorithm Logic

The system uses the 1-Parameter Logistic (1PL) Model from Item Response Theory.

1. Probability Modeling: For every response, we calculate the probability of a correct answer: P(correct) = 1 / (1 + exp(-a * (ability - difficulty))) where a (discrimination) is fixed at 4.0 for a sharper diagnostic curve.

2. Ability Update: We use a stochastic gradient step to update the student's ability estimate ($\theta$): θ_new = θ_old + (step_size / √n) * (response - P(correct)) The step size decreases as the question count n increases, ensuring the estimate converges toward the student's true proficiency.

3. Targeting: The next question is selected by finding the one that provides the maximum Fisher Information at the student's current ability level—essentially choosing questions where the student has roughly a 50% chance of success, providing the most "signal" about their boundaries.

---

🤖 AI Log (Productivity & Challenges)

How AI Tools Were Used

• Architectural Scaffolding: Used LLMs to quickly scaffold the FastAPI project structure and Pydantic models, saving hours of boilerplate setup.
• IRT Implementation: AI assisted in translating mathematical IRT formulas into clean, vectorized-style Python code and explaining the convergence properties of the update rule.
• UI/UX Design: Used AI to generate a cohesive color palette (Teal/Emerald/Amber) and CSS layouts, ensuring a professional look without relying on external UI libraries like Tailwind.
• Data Generation: Assisted in drafting GRE-style questions with specific difficulty rankings and explanations.
• Clerk Integration: AI helped research and implement Clerk authentication for FastAPI (no official SDK) using manual JWT verification with PyJWT.

Challenges Encountered

• LLM JSON Stability: Initial prompts for the study plan occasionally returned markdown formatting inside the JSON. I resolved this by adding strict system instructions and a robust parsing fallback in the llm_service.
• IRT Convergence: Early versions of the algorithm jumped too drastically in difficulty. I introduced a decaying step size (1/√n) to stabilize the ability estimate as the test progresses.
• MongoDB Async Patterns: Ensuring proper async connection handling in FastAPI's lifespan events required careful configuration of the Motor client to prevent connection leaks.
• Clerk + FastAPI: Since Clerk doesn't have an official Python SDK, I implemented manual JWT verification using PyJWT and Clerk's JWKS endpoint for secure token validation.

---

📡 API Documentation

Endpoint	Method	Auth	Description
POST /api/start-session	POST	Optional	Initialize a session and get the first question.
GET /api/next-question/{id}	GET	No	Get the next adaptively-selected question.
POST /api/submit-answer	POST	No	Submit an answer, get feedback, and update ability.
GET /api/study-plan/{id}	GET	No	Generate LLM-powered study plan after completion.
GET /api/session/{id}	GET	No	Retrieve full history and stats for a session.
GET /api/questions/stats	GET	No	View question bank distribution and difficulty metrics.
GET /api/user/history	GET	Required	Get all past test sessions for authenticated user.
DELETE /api/user/sessions/incomplete	DELETE	Required	Remove all incomplete (abandoned) sessions for the user.
GET /api/user/me	GET	Required	Get current user info from Clerk token.

---

Developed as an Internship Assignment for Adaptive Testing Systems.