AgentPrompt

Intelligent prompt optimization service for AI agents with effectiveness evaluation and quality assurance.

⚠️ Beta Release (v0.4.0) - This project is in active development. Core features are stable and tested. Now with effectiveness evaluation, regression detection, and feedback collection.

Features

🎯 Smart Optimization: Four-stage pipeline (static analysis, rule-based transformation, local AI, cloud AI)
🧠 Domain Awareness: Automatic detection of 8 domain categories with tailored optimization strategies
✅ Context Preservation: Validates that optimization maintains original intent and domain terminology
🛡️ Quality Thresholds: Automatic rejection of low-quality optimizations with detailed feedback
📈 Effectiveness Evaluation: Output-based quality scoring that validates actual LLM performance (v0.4.0)
🔍 Regression Detection: Automatic detection and prevention of quality degradation (v0.4.0)
💬 Feedback System: User feedback collection and metric refinement (v0.4.0)
💰 Cost-Benefit Analysis: ROI calculation for optimization efforts (v0.4.0)
🤔 Clarification Engine: Identifies ambiguities and generates targeted questions instead of making assumptions
🔌 Dual Interface: MCP Server for AI clients + REST API for programmatic access
🚀 Zero Dependencies: Core functionality works without external AI services
🏠 Privacy-First: Optional local AI enhancement via Ollama
☁️ Cloud-Ready: Optional cloud AI enhancement (Claude, GPT-4)
📊 Quality Metrics: Quantitative scoring for clarity, completeness, and agent readiness

What's New in v0.4.0

Effectiveness Evaluation

Validates optimization quality by analyzing actual LLM outputs:

const result = await optimize({
  prompt: "Analyze patient lab results for diabetes indicators",
  evaluateEffectiveness: true,
});

// Real-world quality metrics
console.log(result.metrics.output_quality); // 92
console.log(result.metrics.task_completion); // 95
console.log(result.metrics.domain_appropriate); // 94
console.log(result.metrics.cost_efficiency); // 87

Regression Detection

Automatically prevents quality degradation:

const result = await optimize({
  prompt: "Calculate patient risk scores",
});

if (result.rejected) {
  console.log("Optimization rejected:", result.reason);
  console.log("Degraded metrics:", result.regression.degradedMetrics);
  // Original prompt returned automatically
}

Feedback System

Collect user feedback to continuously improve:

const result = await optimize(prompt);

// Submit feedback
await fetch(result.feedbackUrl, {
  method: "POST",
  body: JSON.stringify({
    rating: 5,
    helpful: true,
    preservedIntent: true,
    comments: "Excellent optimization!",
  }),
});

// Feedback automatically refines metrics over time

Cost-Benefit Analysis

Understand the ROI of optimization:

const result = await optimize(prompt);

console.log(result.costBenefit);
// {
//   qualityImprovement: 15.5,
//   timeInvestmentMs: 450,
//   tokenSavings: -20,
//   roi: 8.5
// }

See MIGRATION-v0.4.0.md for complete details.

Quick Start

Automated Setup (Recommended)

# Clone the repository
git clone https://github.com/keyurgolani/agentprompt.git
cd agentprompt

# Run automated setup
npm run setup

# Start development environment with Docker
npm run dev:docker

The setup script will:

Install Node.js dependencies
Create environment configuration files
Set up Docker and Ollama (if available)
Pull required AI models

Manual Setup

# Install dependencies
npm install

# Copy environment configuration
cp .env.example .env.development

# Build the project
npm run build

# Start the server
npm start

Installation

Prerequisites

Node.js 18+ (required)
Docker Desktop (optional, for local AI)
8GB+ RAM (recommended for Ollama models)
10GB+ disk space (for models and dependencies)

MCP Server Installation

Claude Desktop

Add to your Claude Desktop configuration (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "agentprompt": {
      "command": "node",
      "args": ["/path/to/agentprompt/dist/index.js"],
      "env": {
        "OLLAMA_HOST": "http://localhost:11434",
        "USE_LOCAL_AI": "true"
      }
    }
  }
}

Cursor

Add to Cursor's MCP settings (.cursor/mcp.json):

{
  "mcpServers": {
    "agentprompt": {
      "command": "node",
      "args": ["/path/to/agentprompt/dist/index.js"],
      "env": {
        "OLLAMA_HOST": "http://localhost:11434",
        "USE_LOCAL_AI": "true"
      }
    }
  }
}

VSCode

Install the MCP extension and add to your settings:

{
  "mcp.servers": {
    "agentprompt": {
      "command": "node",
      "args": ["/path/to/agentprompt/dist/index.js"],
      "env": {
        "OLLAMA_HOST": "http://localhost:11434",
        "USE_LOCAL_AI": "true"
      }
    }
  }
}

REST API Installation

The REST API can run standalone or alongside the MCP server:

# Enable REST API in .env.development
ENABLE_REST_API=true
REST_API_PORT=3000

# Start the server
npm start

Configuration

Environment Variables

Create .env.development from .env.example:

# Server Configuration
NODE_ENV=development

# REST API Configuration
ENABLE_REST_API=true
REST_API_PORT=3000
API_KEY=your-secure-api-key-here
CORS_ORIGINS=*
RATE_LIMIT_REQUESTS=100
RATE_LIMIT_WINDOW_MS=60000

# Local AI (Ollama)
OLLAMA_HOST=http://localhost:11434
OLLAMA_MODEL=llama3.2:3b
USE_LOCAL_AI=true

# Cloud AI (Optional)
USE_CLOUD_AI=false
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...

# Optimization Settings
MAX_PROMPT_LENGTH=50000
OPTIMIZATION_TIMEOUT_MS=10000
ENABLE_CACHING=true
CACHE_TTL_HOURS=24

# Logging
LOG_LEVEL=info
LOG_FORMAT=pretty

Configuration Options

Variable	Default	Description
`ENABLE_REST_API`	`false`	Enable REST API server
`REST_API_PORT`	`3000`	REST API port
`API_KEY`	-	API key for authentication (optional)
`CORS_ORIGINS`	`*`	Allowed CORS origins
`RATE_LIMIT_REQUESTS`	`100`	Max requests per window
`RATE_LIMIT_WINDOW_MS`	`60000`	Rate limit window (ms)
`OLLAMA_HOST`	`http://localhost:11434`	Ollama service URL
`OLLAMA_MODEL`	`llama3.2:3b`	Ollama model to use
`USE_LOCAL_AI`	`false`	Enable local AI enhancement
`USE_CLOUD_AI`	`false`	Enable cloud AI enhancement
`ANTHROPIC_API_KEY`	-	Anthropic API key
`OPENAI_API_KEY`	-	OpenAI API key
`MAX_PROMPT_LENGTH`	`50000`	Maximum prompt length
`OPTIMIZATION_TIMEOUT_MS`	`10000`	Optimization timeout
`ENABLE_CACHING`	`true`	Enable result caching
`CACHE_TTL_HOURS`	`24`	Cache TTL in hours

Usage

MCP Server

Once configured in your MCP client, AgentPrompt provides three tools:

1. optimize_prompt

Optimizes a prompt through the optimization pipeline:

// In Claude Desktop, Cursor, or VSCode
use_mcp_tool("agentprompt", "optimize_prompt", {
  prompt: "Write a function to sort an array",
  optimization_level: "balanced", // "fast", "balanced", or "thorough"
  use_local_ai: true,
  use_cloud_ai: false,
});

Response:

{
  "optimized_prompt": "<role>You are a coding assistant...</role>...",
  "improvements": [
    {
      "category": "structure",
      "description": "Added role definition",
      "impact": "high"
    }
  ],
  "clarification_questions": [
    {
      "question": "What programming language should the function be written in?",
      "category": "context",
      "priority": "critical",
      "rationale": "Language choice affects implementation approach"
    }
  ],
  "metrics": {
    "clarity_score": 85,
    "completeness_score": 75,
    "agent_readiness_score": 80,
    "processing_time_ms": 45
  }
}

2. analyze_prompt

Analyzes a prompt without modifying it:

use_mcp_tool("agentprompt", "analyze_prompt", {
  prompt: "Process the data",
  check_types: ["clarity", "completeness"], // optional
});

Response:

{
  "issues": [
    {
      "type": "vague_objective",
      "severity": "warning",
      "message": "Objective is unclear",
      "suggestion": "Specify what processing should be done"
    }
  ],
  "strengths": [],
  "clarification_questions": [
    {
      "question": "What specific processing should be performed on the data?",
      "category": "objective",
      "priority": "critical"
    }
  ],
  "overall_quality": "fair"
}

3. compare_prompts

Compares two prompts quantitatively:

use_mcp_tool("agentprompt", "compare_prompts", {
  original_prompt: "Write a function",
  optimized_prompt: "<role>You are a coding assistant...</role>...",
});

REST API

The REST API provides HTTP endpoints for programmatic access.

Authentication

If API_KEY is configured, include it in requests:

curl -H "Authorization: Bearer your-api-key" \
     -H "Content-Type: application/json" \
     -X POST http://localhost:3000/api/optimize \
     -d '{"prompt": "Write a function"}'

Endpoints

POST /api/optimize - Optimize a prompt

curl -X POST http://localhost:3000/api/optimize \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "prompt": "Write a function to sort an array",
    "optimization_level": "balanced",
    "use_local_ai": true,
    "use_cloud_ai": false
  }'

POST /api/analyze - Analyze a prompt

curl -X POST http://localhost:3000/api/analyze \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "prompt": "Process the data",
    "check_types": ["clarity", "completeness"]
  }'

POST /api/compare - Compare two prompts

curl -X POST http://localhost:3000/api/compare \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer your-api-key" \
  -d '{
    "original_prompt": "Write a function",
    "optimized_prompt": "<role>You are a coding assistant...</role>..."
  }'

GET /api/health - Health check

curl http://localhost:3000/api/health

See API.md for complete API documentation.

Documentation

Core Documentation

API.md - Complete API reference for MCP and REST interfaces
EFFECTIVENESS-EVALUATION.md - Output-based quality scoring (v0.4.0+)
REGRESSION-DETECTION.md - Quality degradation detection (v0.4.0+)
FEEDBACK-SYSTEM.md - User feedback collection (v0.4.0+)
MIGRATION-v0.4.0.md - Migration guide from v0.3.0 to v0.4.0
DOMAIN-DETECTION.md - Domain detection system and strategies (v0.3.0+)
CONTEXT-PRESERVATION.md - Context preservation validation (v0.3.0+)
MIGRATION-v0.3.0.md - Migration guide from v0.2.0 to v0.3.0
RULES.md - Documentation of all optimization rules
CLARIFICATIONS.md - Clarification engine behavior and examples
DEVELOPMENT.md - Development guide, tooling, and optimization context
DOCKER.md - Docker setup for development and testing

Comparative Study & Enhancement

Comparative Study System - Comprehensive guide to the comparative study system
Optimization Module - AI enhancement validation and optimization pipeline
Rules Module - All optimization rules and rule engine

Development

Development Scripts

# Docker development (recommended)
npm run dev:docker          # Start development environment
npm run dev:stop            # Stop development environment
npm run dev:logs            # View application logs
npm run dev:restart         # Restart application service
npm run dev:shell           # Open shell in container
npm run dev:pull-models     # Pull Ollama models

# Local development
npm run dev                 # Start with hot reload
npm run build               # Compile TypeScript
npm start                   # Run production build

# Code quality
npm run lint                # Run ESLint
npm run format              # Format with Prettier

Project Structure

agentprompt-pro/
├── src/
│   ├── server/            # MCP and REST server implementations
│   │   ├── mcp-server.ts
│   │   ├── rest-api.ts
│   │   └── tools/         # Tool handlers
│   ├── optimization/      # Core optimization engine
│   │   ├── pipeline.ts
│   │   ├── analyzer.ts
│   │   ├── transformer.ts
│   │   ├── local-ai-enhancer.ts
│   │   └── cloud-ai-enhancer.ts
│   ├── clarification/     # Clarification engine
│   │   └── clarification-engine.ts
│   ├── rules/             # Optimization rules
│   │   ├── structural-rules.ts
│   │   ├── clarity-rules.ts
│   │   └── agent-rules.ts
│   ├── utils/             # Utilities and helpers
│   └── types/             # TypeScript type definitions
├── tests/
│   ├── unit/              # Unit tests
│   ├── integration/       # Integration tests
│   ├── e2e/               # End-to-end tests
│   └── helpers/           # Test utilities
├── scripts/               # Development and test scripts
└── docs/                  # Additional documentation

Testing

Running Tests

# Run all tests
npm test

# Run specific test suites
npm run test:unit
npm run test:integration
npm run test:e2e

# Run with coverage
npm run test:coverage

# Watch mode
npm run test:watch

# Docker-based testing
npm run test:docker
npm run test:docker:unit
npm run test:docker:integration
npm run test:docker:e2e

Test Coverage

The project maintains 90%+ test coverage across:

Unit tests for rules, analyzers, and utilities
Integration tests for pipeline and tool handlers
E2E tests for MCP and REST interfaces

See DOCKER.md for details.

Requirements

Node.js 18+
Docker Desktop (optional, for local AI)
8GB+ RAM (for Ollama models)
10GB+ disk space

License

MIT

Status

⚠️ Beta Release (v0.4.0) - Core functionality implemented and tested with 90%+ coverage. Now featuring effectiveness evaluation, regression detection, feedback collection, and cost-benefit analysis. See CHANGELOG.md for details.

Support

Issues: GitHub Issues
Documentation: See docs/ directory
Examples: See API.md and CLARIFICATIONS.md

Contributing

Contributions are welcome! Please read CONTRIBUTING.md for guidelines on how to contribute to this project.

Acknowledgments

Built with @modelcontextprotocol/sdk
Powered by Ollama for local AI
Supports Anthropic Claude and OpenAI GPT-4

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
.github		.github
.husky		.husky
docs		docs
scripts		scripts
src		src
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.env.test.bootstrap		.env.test.bootstrap
.eslintrc.json		.eslintrc.json
.gitignore		.gitignore
.prettierignore		.prettierignore
.prettierrc.json		.prettierrc.json
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
Dockerfile.dev		Dockerfile.dev
Dockerfile.test		Dockerfile.test
LICENSE		LICENSE
README.md		README.md
docker-compose.dev.yml		docker-compose.dev.yml
docker-compose.test.yml		docker-compose.test.yml
docker-compose.yml		docker-compose.yml
package-lock.json		package-lock.json
package.json		package.json
tsconfig.build.json		tsconfig.build.json
tsconfig.json		tsconfig.json
vitest.config.ts		vitest.config.ts
vitest.e2e.config.ts		vitest.e2e.config.ts

Uh oh!

License

keyurgolani/AgentPrompt

Folders and files

Latest commit

History

Repository files navigation

AgentPrompt

Features

Table of Contents

What's New in v0.4.0

Effectiveness Evaluation

Regression Detection

Feedback System

Cost-Benefit Analysis

Quick Start

Automated Setup (Recommended)

Manual Setup

Installation

Prerequisites

MCP Server Installation

Claude Desktop

Cursor

VSCode

REST API Installation

Configuration

Environment Variables

Configuration Options

Usage

MCP Server

1. optimize_prompt

2. analyze_prompt

3. compare_prompts

REST API

Authentication

Endpoints

Documentation

Core Documentation

Comparative Study & Enhancement

Development

Development Scripts

Project Structure

Testing

Running Tests

Test Coverage

Requirements

License

Status

Support

Contributing

Acknowledgments

About

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Sponsor this project

Uh oh!

Packages 0

Languages

Packages