CLI Compare

   ╭─────────────────╮
   │  ┌───┐  ┌───┐   │
   │  │>_ │vs│ _<│   │
   │  └───┘  └───┘   │
   │    CLI COMPARE  │
   ╰─────────────────╯

A Windows Node.js/React application to compare multiple AI coding CLI tools by running them in parallel WSL/tmux sessions against git worktrees. Run the same prompt against different coding assistants and compare their outputs and code changes.

License

This project is licensed under the MIT License - see the LICENSE file for details.

The MIT License is recommended because it:

Is simple and permissive, allowing commercial and private use
Has minimal restrictions on redistribution
Is widely recognized and compatible with most other licenses
Encourages adoption and contribution from the community

Supported CLI Tools

CLI Tool	Description
Claude Code	Anthropic's official CLI for Claude (Opus 4.5, Sonnet 4, etc.)
Cursor	AI-first code editor
Aider	AI pair programming in your terminal
GitHub Copilot	AI pair programmer by GitHub
Cody	AI coding assistant by Sourcegraph
Continue	Open-source AI code assistant
OpenCode	Terminal-based AI coding assistant

Features

Core Comparison Engine

Parallel CLI Execution: Run multiple AI coding CLIs (Claude, Copilot, Aider, etc.) side-by-side on the same task
Isolated Git Worktrees: Each CLI tool operates in its own git worktree for complete isolation
Real-time Terminal Output: WebSocket-powered live terminal streaming via xterm.js
Side-by-side Diff View: Compare file changes across tools with synchronized scrolling
Change Analysis: Visual indicators showing identical vs. different changes across runs

Repository Management

Local Repository Support: Add existing git repositories from your system
GitHub Cloning: Clone repositories directly from GitHub URLs
Automatic Path Conversion: Seamless Windows ↔ WSL path handling
Branch Selection: Choose which branch to use as the base for evaluations
Multiple Repositories: Manage and switch between multiple codebases

Evaluation System

Evaluation Templates: Create reusable evaluation setups with selected CLI tools
Session Management: Run multiple sessions per evaluation with different prompts
Parallel Runs: Execute all selected CLI tools simultaneously
Status Tracking: Monitor run states (pending, starting, running, waiting for input, stopped, failed)
Bulk Operations: Start, stop, or reset all runs at once

CLI Configuration

Per-CLI Settings: Configure model, permissions, and arguments for each CLI tool
Model Selection: Choose from available models per CLI (Claude Opus/Sonnet, GPT-4, etc.)
Tool Permissions: Fine-grained control over file, execution, network, and git operations
Quick Arguments: Pre-defined argument shortcuts for common CLI flags
Additional Arguments: Custom CLI arguments for advanced use cases
Configuration Snapshots: Preserve exact configuration used for each run

Interactive Session Control

Input Handling: Send text input to running CLI sessions
Quick Actions: One-click buttons for common responses (y/n, Enter, Ctrl+C, arrows)
Prompt Detection: Automatic detection of yes/no, permission, password prompts
Auto-response: Claude trust folder prompts are automatically confirmed
Session Reset: Kill tmux session and reset worktree to base branch state
Input History: Track all inputs sent during a session

Prompt Library

File-based Prompts: Store and organize prompts as files in a configurable directory
Prompt Collections: Group related prompts into reusable collections
Collection Association: Link prompt collections to evaluations
In-app Editing: Browse, view, and edit prompt files directly

Diff & Comparison

File Change Tracking: See which files each CLI tool modified
Unified Diff View: Standard git diff format with syntax highlighting
Multi-run Comparison: Compare changes between any two runs or against original
Change Statistics: Line additions/deletions per file and total
Status Indicators: Visual markers for same/different/unique changes across runs

Commit & Push

Commit Changes: Commit modifications made by a CLI run
Push to Branch: Push worktree changes to a new branch
Create Pull Request: Open PR directly from the app via GitHub CLI

System Management

Prerequisite Checker: Validates WSL, tmux, git, gh installation on startup
Guided Installation: Install missing dependencies through the UI with sudo support
GitHub OAuth: Device code authentication flow for gh CLI
Settings Configuration: Configure repository and prompt library directories

Prerequisites

Required

Node.js 18+ - JavaScript runtime (download)
WSL2 - Windows Subsystem for Linux with a Linux distribution (Ubuntu recommended)
- Install via: wsl --install in PowerShell (Admin)

Auto-Installed Dependencies

The following dependencies will be auto-detected and can be installed through the app's setup wizard:

tmux - Terminal multiplexer for managing CLI sessions
git - Version control system
GitHub CLI (gh) - For repository operations and authentication

AI CLI Tools (Install at least one)

The app compares these CLI coding assistants - install the ones you want to evaluate:

CLI Tool	Installation	Notes
Claude Code	`npm install -g @anthropic-ai/claude-code`	Requires Anthropic API key
GitHub Copilot CLI	`gh extension install github/gh-copilot`	Requires GitHub Copilot subscription
Aider	`pip install aider-chat`	Supports multiple LLM providers
Cody	`npm install -g @sourcegraph/cody`	Sourcegraph's AI assistant
OpenCode	`go install github.com/opencode-ai/opencode@latest`	Terminal-based AI assistant
Continue	VS Code/JetBrains extension	Open-source AI assistant
Cursor	cursor.sh	AI-first code editor

On first launch, the app checks for prerequisites and guides you through installation if any are missing. GitHub authentication uses the device code flow for secure sign-in.

Installation

Clone the repository

git clone https://github.com/your-username/coding-cli-what-if.git
cd coding-cli-what-if

Install dependencies
```
npm install
```
Build the shared package
```
npm run build -w @cli-compare/shared
```
Build all packages (for production)
```
npm run build
```

Running the Application

Development Mode

Start both frontend and backend with hot reload:

npm run dev

Or run them separately:

# Backend (runs on port 3001)
npm run dev:backend

# Frontend (runs on port 5173)
npm run dev:frontend

Open http://localhost:5173 in your browser.

Production Mode

Build all packages:
```
npm run build
```
Start the backend server:
```
npm start -w @cli-compare/backend
```

Configuration

Initial Setup

On first launch, the app checks for prerequisites and guides you through installation:

WSL2 with a Linux distribution
tmux - Terminal multiplexer
git - Version control
GitHub CLI (gh) - For repository operations

App Settings

Configure via the Settings modal (gear icon in the header):

Repository Base Directory: Windows path where cloned repositories will be stored (e.g., C:\Repos)
Prompt Library Directory: Windows path where prompt files are stored (e.g., C:\Prompts)

CLI Configurations

Create configurations for each CLI tool:

Go to Configurations page
Click New Configuration
Select the CLI tool and model
Configure tool permissions
Save

Usage

Creating an Evaluation

Add a Repository: Go to Repositories → Add or Clone a git repository
Create Evaluation: Go to Evaluations → New Evaluation
- Select a repository and branch
- Select which CLI tools to include
Create a Session: From the Evaluations list, create a new session
- Enter your prompt (the task for the AI)
- A session contains one run per CLI tool configured in the evaluation
Open Session: Click on a session to access the terminal view
Start Runs: Start individual runs or all at once
Monitor Progress: Watch real-time terminal output for each CLI tool
Compare Results: Use the diff view to compare file changes across runs

Run Controls

Start/Stop: Control individual runs or all at once
Send Input: Type in the input field and press Enter
Quick Actions: Use buttons for common inputs (y, n, Enter, Ctrl+C, arrows)
Reset: Kill tmux session and reset worktree to base branch
Auto-response: Claude trust folder prompts are automatically confirmed

Project Structure

coding-cli-what-if/
├── package.json                  # Root workspace
├── apps/
│   ├── backend/                  # Express + TypeScript API
│   │   ├── src/
│   │   │   ├── index.ts          # Entry point
│   │   │   ├── db/               # SQLite database
│   │   │   ├── routes/           # API routes
│   │   │   ├── services/         # Business logic
│   │   │   │   ├── wsl.service.ts
│   │   │   │   ├── tmux.service.ts
│   │   │   │   ├── worktree.service.ts
│   │   │   │   └── diff.service.ts
│   │   │   ├── models/           # Database models
│   │   │   └── websocket/        # WebSocket handlers
│   │   └── data/                 # SQLite database files
│   │
│   └── frontend/                 # React + Vite
│       └── src/
│           ├── pages/            # Dashboard, Repos, Runs, Compare
│           ├── components/       # Terminal, etc.
│           ├── store/            # Zustand stores
│           └── services/         # API clients
│
└── packages/
    └── shared/                   # Shared types & constants

API Endpoints

Repositories

GET /api/repos - List all repos
POST /api/repos - Create a repo
GET /api/repos/:id - Get a repo
PUT /api/repos/:id - Update a repo
DELETE /api/repos/:id - Delete a repo

Evaluations

GET /api/evaluations - List all evaluations
POST /api/evaluations - Create an evaluation
GET /api/evaluations/:id - Get an evaluation with sessions
PATCH /api/evaluations/:id - Update an evaluation
DELETE /api/evaluations/:id - Delete an evaluation (cleans up worktrees/tmux)
POST /api/evaluations/:id/sessions - Create a new session for an evaluation

Sessions

GET /api/sessions/:id - Get a session with runs
POST /api/sessions/:id/reset - Reset all runs in a session
DELETE /api/sessions/:id - Delete a session

Session Runs

GET /api/runs/:id - Get a run
POST /api/runs/:id/start - Start a run
POST /api/runs/:id/stop - Stop a run
POST /api/runs/:id/reset - Reset a run (kill tmux, reset worktree)
POST /api/runs/:id/input - Send input to a run
POST /api/runs/:id/interrupt - Send Ctrl+C to a run
POST /api/runs/:id/special-key - Send special key (arrows, Home, End, Backspace)
GET /api/runs/:id/history - Get input history

Diff

GET /api/diff/runs/:runId/changes - Get changed files for a run
GET /api/diff/runs/:runId/file?path=... - Get file diff
GET /api/diff/sessions/:sessionId/all-files - Get all changed files across runs

System

GET /api/system/status - Check system prerequisites (WSL, tmux, git, gh, auth status)
POST /api/system/install/:package - Install a package (tmux, git, gh) with sudo password
GET /api/system/install/:installId/progress - Get installation progress
POST /api/system/gh/auth - Start GitHub device code authentication
GET /api/system/gh/auth/:sessionId - Get GitHub auth session status

WebSocket

Connect to /ws for real-time updates:

// Subscribe to a session (subscribes to all runs in the session)
ws.send(JSON.stringify({ type: 'subscribe_session', sessionId: 1 }));

// Subscribe to a specific run
ws.send(JSON.stringify({ type: 'subscribe_run', runId: 1 }));

// Receive terminal output for a run
// { type: 'terminal_output', sessionId: 1, runId: 1, data: '...' }

// Receive run status updates
// { type: 'run_status', runId: 1, status: 'running' }

// Receive prompt detection (waiting for input)
// { type: 'prompt_detected', runId: 1, promptType: 'yes_no', options: ['y', 'n'] }

Tech Stack

Backend: Node.js, Express, TypeScript, better-sqlite3, ws
Frontend: React, Vite, TypeScript, Tailwind CSS, Zustand, xterm.js
Infrastructure: WSL2, tmux, git worktrees

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
apps		apps
packages/shared		packages/shared
.gitignore		.gitignore
README.md		README.md
package-lock.json		package-lock.json
package.json		package.json

amolchanov/coding-cli-comparer

Folders and files

Latest commit

History

Repository files navigation