InferBridge

Local AI for Intel hardware. Powered by OpenVINO GenAI.

InferBridge turns Intel Windows PCs into local AI workstations. It provides a Windows-first, OpenAI-compatible server with streaming chat, local vision chat, text embeddings, model lifecycle and recovery workflows, secure Hugging Face access, context-budget visibility, CPU/GPU/NPU device targeting, System Doctor diagnostics, hardware benchmarking, a built-in browser UI, and a deterministic mock engine.

OpenVINO GenAI remains the inference runtime. InferBridge does not replace OpenVINO, claim ownership of it, or imply affiliation with Intel. The project does not require Docker, cloud inference, Electron, or a Node frontend toolchain.

Video walkthrough

Watch the InferBridge walkthrough

Project and release status

Current stable release: 0.9.5. See 0.9.5 for the full record. It brings transactional model preparation, strict converted-artifact validation, working packaged conversion, reliable install-over-existing and uninstall flows, uninterrupted device switching, model preparation recovery, stage-aware preparation watchdogs, secure Hugging Face access, and context-budget visibility to the stable channel.

• Windows installer
• Portable ZIP
• SHA-256 checksums

There is no pre-release newer than the current stable release. Versions 0.9.0 through 0.9.3 were published only as betas and are superseded by 0.9.5. Beta builds ship ahead of the next stable release and are marked pre-release on GitHub.

Releases from v0.7.0 onward use the canonical Quazmoz/InferBridge repository and InferBridge-<version> artifact names. Releases through v0.6.3 were published before the rename, and their repository links and artifact filenames intentionally retain the former product identity.

Validation scope: Mock-mode tests validate API, UI, packaging, lifecycle, and state contracts. Real CPU, GPU, and NPU claims require the Windows certification harness on suitable hardware with mock mode disabled. An unsigned development installer is not a signed production release.

The retained 0.6.1 hardware evidence comes from one Intel Core Ultra 9 185H laptop running Windows 11 build 26200 and OpenVINO 2026.2.1. It is not a claim that every Intel system, model, precision, driver, or OpenVINO release will behave the same way, and no later release has been recertified on hardware.

Signing status: No published InferBridge installer or portable ZIP is Authenticode-signed. Windows reports an unknown publisher and SmartScreen may warn on first launch. Verify downloads against the published SHA-256 checksums.

Download, install, choose a model, and chat

1. Download a versioned installer or portable ZIP.
2. Run InferBridge.exe from the installed shortcut or extracted directory.
3. Review the hardware and OpenVINO system scan.
4. Confirm the available Intel CPU, GPU, or NPU target.
5. Accept the conservative model recommendation or choose another compatible model.
6. Confirm model license, disk, and warning information.
7. Follow the download, conversion, validation, compilation, loading, and benchmark stages.
8. Continue to chat or copy the OpenAI-compatible endpoint for another client.

The desktop launcher binds to 127.0.0.1, selects an available local port, waits for liveness and readiness, opens the browser, and prevents duplicate instances. Model weights are downloaded separately and are not included in the base installer.

The first model setup can take significant time. NPU support depends on the actual Intel platform, driver, model, precision, and OpenVINO release. A requested target is not proof of execution. InferBridge reports the actual device from successful measured generation when the runtime exposes that evidence.

Beta Testing & Feedback

Found a bug? Have a feature request? Testing InferBridge and want to send general beta feedback? No GitHub experience is required.

Easy feedback: Send InferBridge feedback using the user-friendly support form for bugs, feature requests, and beta feedback.

Developer / technical discussion: GitHub Issues remain available for contributors, advanced technical reports, and public engineering discussion.

For troubleshooting details, use Tray → Copy → Diagnostics or System Doctor → Copy support report, then paste the privacy-safe output into either destination. InferBridge does not automatically upload diagnostics.

Installation options

Windows installer

The Inno Setup package installs per-user under:

%LOCALAPPDATA%\Programs\InferBridge

It creates a Start Menu shortcut, can create an optional desktop shortcut, and preserves mutable data by default during upgrades and uninstall.

See the Windows installer guide.

Portable ZIP

Extract the complete ZIP to a writable directory and run InferBridge.exe. Portable mode stores mutable data under a sibling data directory and does not create registry or Start Menu entries.

See the portable package guide.

Source setup on Windows

git clone https://github.com/Quazmoz/InferBridge.git
cd InferBridge
.\setup.bat
.\start_server.bat --mock

Start a real converted model on CPU:

.\start_server.bat `
  --model tinyllama-1.1b-chat-fp16 `
  --device CPU `
  --auto-convert

Use GPU, NPU, or AUTO only when OpenVINO reports the target:

.\start_server.bat --check-devices

See QUICKSTART.md for the complete setup flow.

Visual preview

Main chat interface

System Doctor hardware diagnostics

Settings and system information

First-run model flow

Light theme

Responsive layout

Documentation

Start and operate

• Quick start
• First-run guide
• Windows installer
• Portable package
• Windows setup
• Experimental Linux overview
• Experimental Ubuntu setup
• Experimental Fedora setup

Architecture and APIs

• Desktop architecture
• Desktop onboarding API
• API contract
• Local vision chat
• Open WebUI guide
• Open WebUI and n8n integrations
• Device support
• Verified Model Library
• Model preparation cancellation
• Model preparation recovery
• Context budget visibility

Data, diagnostics, and support

• Mutable data paths
• Privacy-safe diagnostics
• Desktop troubleshooting
• Known issues
• Compatibility matrix
• Windows hardware certification

Packaging and release

• Packaging and release overview
• Production release process
• Code signing
• Versioning
• Update policy
• Upgrade and rollback
• Third-party licenses

Rename and compatibility

• InferBridge migration and repository rename
• InferBridge identity inventory

Release notes

• 0.9.5 — current stable release

Implemented capabilities

Inference and OpenAI compatibility

• OpenVINO GenAI text generation on CPU, GPU, NPU, AUTO, and advanced composite expressions such as AUTO:NPU,GPU,CPU, MULTI:NPU,GPU,CPU, and HETERO:NPU,GPU,CPU
• POST /v1/chat/completions with streaming and non-streaming responses
• POST /v1/responses with streaming and non-streaming output
• POST /v1/embeddings for registered embedding models such as bge-small-en-v1.5
• OpenAI-compatible image content parts for local vision-language models
• Stop sequences, seed, temperature, top-p, token limits, tool-call compatibility, and structured-output fields
• Safe-by-default Hugging Face conversion with remote code disabled unless explicitly enabled for a reviewed model

Model and device operations

• Maintained model catalog plus custom Hugging Face registration
• Verified Model Library with retained official evidence and local benchmark evidence
• System Doctor checks for runtime, devices, routing, memory, disk, model readiness, and NPU state
• Transactional conversion staging, strict artifact validation, bounded backup and rollback, and incomplete-output protection
• Sanitized progress, operation-scoped cancellation, cross-process preparation locks, and serialized lifecycle operations
• Recovery state with resume, failed-stage retry, fresh-download restart, and incomplete-output cleanup actions
• Stage-aware download, conversion, finalization, loading, and compilation watchdogs that distinguish slow progress from a stalled operation
• Secure Hugging Face token storage with Windows DPAPI, gated-model access preflight, and environment variables retained as an advanced fallback
• Optional conversion when loading a missing model
• OpenVINO hardware discovery and NPU readiness checks
• Uninterrupted device switching that keeps the working engine available until replacement handoff
• Local benchmark persistence and conservative target recommendations

Windows desktop distribution

• Hidden-console launcher with loopback binding
• Per-user single-instance lock with live nonce verification
• Bounded liveness and readiness polling
• Safe port fallback and duplicate-instance prevention
• Startup dialogs, system tray integration, and rotating logs
• Privacy-safe diagnostic ZIP export
• Installed and portable writable-path strategies
• Versioned onboarding state with corruption recovery
• Connection examples for the OpenAI Python SDK, environment variables, Open WebUI, and n8n
• PyInstaller executable, portable ZIP, Inno Setup installer, checksums, release manifests, and Authenticode signing gates

Browser UI

• Responsive local chat with browser-stored conversation history
• Light and dark themes
• Model selection, device targeting, load, unload, deletion, conversion, and cancellation controls
• Model preparation recovery with reusable-cache status, recommended actions, and sanitized failure details
• Exact tokenizer-backed context-budget preflight with whole-turn omission visibility and output-limit actions
• Secure Hugging Face access settings and gated-model preflight without storing tokens in browser localStorage
• Active operation progress and queue visibility
• System Doctor diagnostics and privacy-safe support report copying
• Hardware Benchmark comparisons with TTFT and tokens-per-second measurements
• Dependency-free Markdown rendering, syntax highlighting, and code copying
• Token, latency, RAM, disk, and request metrics
• Accessible first-run onboarding with visible progress and keyboard navigation

Data paths and compatibility

Installed mode uses:

%LOCALAPPDATA%\InferBridge

Portable mode uses:

<portable directory>\data

Configuration, logs, models, Hugging Face cache, OpenVINO compiled cache, benchmarks, diagnostics, and onboarding state are separated. Normal upgrades and uninstall preserve mutable data unless the user explicitly requests removal.

InferBridge 0.7.0 and later recognize the legacy %LOCALAPPDATA%\OpenVINOWindowsLLM data directory and do not automatically move or merge multi-gigabyte model data. See DATA_PATHS.md and UPGRADE_ROLLBACK.md.

Existing environment variables remain supported:

OV_LLM_DATA_DIR
OV_LLM_MODELS_FILE
OV_LLM_MODELS_DIR
OV_LLM_CACHE_DIR
OV_LLM_BENCHMARK_RESULTS
OV_LLM_API_KEY
OV_LLM_DEVICE
OV_LLM_MODEL
OV_LLM_MOCK

The OV_LLM_* prefix, ov-llm commands, Python distribution name openvino-windows-llm, and internal .ovllm-* metadata names are retained for backward compatibility. New command aliases are:

inferbridge
inferbridge-desktop

API overview

GET    /                         Built-in InferBridge UI
GET    /health                   Runtime and lifecycle summary
GET    /health/live              Liveness probe
GET    /health/ready             Readiness probe

GET    /v1/models                     OpenAI-style model list
POST   /v1/chat/completions           Streaming or non-streaming chat
POST   /v1/chat/context-budget        Exact context-window preflight
POST   /v1/responses                  Responses API
POST   /v1/embeddings                 Float or base64 embeddings

POST   /v1/models/register            Register a custom model
POST   /v1/models/convert             Convert a model
POST   /v1/models/load                Load a model
POST   /v1/models/unload              Unload a model
POST   /v1/models/delete              Delete an unloaded model
POST   /v1/models/cancel              Cancel the current preparation operation
GET    /v1/models/recovery/{model_id} Get sanitized recovery details
POST   /v1/models/recovery/action     Apply a scoped recovery action

GET    /v1/huggingface/status         Read masked Hugging Face access status
POST   /v1/huggingface/token          Validate and securely store a token
DELETE /v1/huggingface/token          Remove the securely stored token
POST   /v1/huggingface/test           Test configured Hugging Face access
POST   /v1/huggingface/preflight      Check model access before conversion

GET    /v1/devices                    OpenVINO device discovery
GET    /v1/system/status         Telemetry, lifecycle, metrics, events
GET    /v1/model-library         Curated model library
POST   /v1/model-library/refresh Refresh approved model definitions
POST   /v1/benchmarks/run        Run local hardware benchmarks
GET    /v1/benchmarks            List saved benchmark runs

State-changing routes follow the existing API-key policy when OV_LLM_API_KEY is configured. Connection examples return placeholders rather than exposing the configured key.

Configuration example

$env:OV_LLM_HOST = "127.0.0.1"
$env:OV_LLM_PORT = "8000"
$env:OV_LLM_DEVICE = "CPU"
$env:OV_LLM_MODEL = "tinyllama-1.1b-chat-fp16"
$env:OV_LLM_AUTO_CONVERT = "1"
$env:OV_LLM_API_KEY = ""
$env:OV_LLM_CORS_ORIGINS = ""
$env:OV_LLM_RATE_LIMIT = "0"
$env:OV_LLM_MOCK = ""
$env:HF_TOKEN = ""

Keep the host at 127.0.0.1 unless LAN access is deliberate, authenticated, firewall-restricted, and limited to a trusted private network. InferBridge is not a hardened public internet gateway.

Device behavior

Simple targets:

CPU
GPU
NPU
AUTO

Advanced examples:

AUTO:NPU,GPU,CPU
MULTI:NPU,GPU,CPU
HETERO:NPU,GPU,CPU

Composite routing is experimental and does not guarantee faster generation. OpenVINO discovery is the source of truth. Real support claims require the Windows certification harness.

Build and validate a Windows distribution

.\scripts\build_windows_distribution.ps1
.\scripts\smoke_test_packaged.ps1 -DistributionPath .\dist\InferBridge

For the production release process, use scripts\build_release.ps1 and follow docs/RELEASE_PROCESS.md. The release pipeline produces versioned InferBridge artifacts, SHA-256 checksums, release metadata, third-party license inventories, and optional Authenticode signatures.

The packaged mock smoke test validates startup, readiness, UI, onboarding, model preparation, benchmarks, Chat Completions, Responses API, streaming, portable paths, and controlled shutdown. It does not prove real CPU, GPU, NPU, model conversion, installer upgrade, Authenticode trust, or SmartScreen reputation.

Development and testing

pip install -e .[dev]
ruff check .
ruff format --check .
pytest

External mock contract validation:

OV_LLM_MOCK=1 python -m app.server --mock --host 127.0.0.1 --port 8123 --model tinyllama-1.1b-chat-fp16
python scripts/validate_api_contract.py --base-url http://127.0.0.1:8123 --profile full --expect-mock --include-embeddings --run-benchmark --exercise-lifecycle

Real Windows hardware validation:

.\scripts\validate_windows.ps1

Experimental Linux

git clone https://github.com/Quazmoz/InferBridge.git
cd InferBridge
chmod +x setup.sh start_server.sh setup/*.sh setup/linux/*.sh
./setup.sh --minimal
./start_server.sh --mock
./start_server.sh --model tinyllama-1.1b-chat-fp16 --device CPU

Linux GPU and NPU support remains driver-dependent and experimental. InferBridge remains Windows-first.

Security and privacy

• Loopback is the safe default.
• State-changing routes can require an API key.
• Cross-origin browser access is opt-in.
• Request bodies and image inputs have explicit limits.
• Progress, logs, diagnostics, and user-facing errors sanitize secrets and private paths.
• Diagnostics exclude prompts, chat history, model weights, source images, keys, and certificates.
• Model and conversion lifecycle operations are serialized and bounded.
• Incomplete conversion output is not presented as a valid model.
• Recovery and destructive cleanup refuse symbolic links, Windows junctions, unexpected files, and paths outside configured storage.
• User-supplied Hugging Face tokens use Windows DPAPI storage, are never returned by the API, and are not stored in browser localStorage.
• Certificates, private keys, passwords, tokens, and signing secrets must never enter the repository.

Conversation history remains in browser localStorage and is not persisted by the server.

Contributing

See CONTRIBUTING.md. Use the canonical repository for new clones and links:

https://github.com/Quazmoz/InferBridge

License

MIT. See LICENSE.