Skip to content

feat: add OpenRouter as LLM and embedding provider#56

Open
oglenyaboss wants to merge 3 commits intorepowise-dev:mainfrom
oglenyaboss:feat/openrouter-provider
Open

feat: add OpenRouter as LLM and embedding provider#56
oglenyaboss wants to merge 3 commits intorepowise-dev:mainfrom
oglenyaboss:feat/openrouter-provider

Conversation

@oglenyaboss
Copy link
Copy Markdown

@oglenyaboss oglenyaboss commented Apr 8, 2026

Summary

  • Add OpenRouter as a first-class LLM and embedding provider, enabling access to 200+ models through a single OPENROUTER_API_KEY
  • Both repowise init (doc generation) and repowise serve (chat/search) work with OpenRouter out of the box
  • No new pip dependency — uses the existing openai package (OpenAI-compatible API)

LLM Provider

  • OpenRouterProvider with generate() + stream_chat() support
  • Default model: anthropic/claude-sonnet-4.6
  • Sets recommended HTTP-Referer and X-Title headers for OpenRouter dashboard tracking
  • Rate limits: 60 RPM / 200K TPM (conservative defaults, user-overridable)

Embedding Provider

  • OpenRouterEmbedder for semantic search and chat RAG
  • Default model: google/gemini-embedding-001 (768 dims)
  • One API key covers both LLM and embeddings

Integration Points

  • Registered in both LLM and embedding registries (lazy import)
  • CLI auto-detection from OPENROUTER_API_KEY env var
  • Interactive provider/embedder selection in repowise init and repowise serve
  • Server provider catalog updated for web UI

Known Limitations

  • Cost tracking disabled for OpenRouter: since it proxies 200+ models with varying prices, the fallback pricing would show inflated numbers. Users should check the OpenRouter dashboard for actual costs. A future PR could fetch real pricing via the /api/v1/models endpoint.
  • Chat and embeddings not fully tested — LLM generation was tested end-to-end on a real 800+ file project using qwen/qwen3.6-plus via OpenRouter. Chat and embedding functionality was not integration-tested but follows the same OpenAI-compatible patterns as the existing OpenAI provider.

Test plan

  • 13 unit tests for OpenRouterProvider (construction, generation, error mapping, headers)
  • Registry test updated (builtin count 6 → 7, openrouter in list)
  • Integration test added (skipped without OPENROUTER_API_KEY)
  • Manual end-to-end test: repowise init with OpenRouter on a real project (328 wiki pages generated successfully)
  • Chat via web UI with OpenRouter as active provider
  • Embedding search with OpenRouterEmbedder

Add OpenRouter as a first-class provider, enabling access to 200+ models
(Claude, GPT, Gemini, Llama, Qwen, etc.) through a single API key.

LLM provider:
- New OpenRouterProvider using OpenAI-compatible endpoint
- Supports generate() and stream_chat() (ChatProvider protocol)
- Sets recommended HTTP-Referer and X-Title headers
- Default model: anthropic/claude-sonnet-4.6
- Rate limits: 60 RPM / 200K TPM
- Cost tracking intentionally disabled (OpenRouter proxies models
  with varying prices — users should check the OpenRouter dashboard)

Embedding provider:
- New OpenRouterEmbedder for vector search and chat
- Default model: google/gemini-embedding-001 (768 dims)
- One OPENROUTER_API_KEY covers both LLM and embeddings

Integration:
- Registered in LLM and embedding registries (lazy import)
- CLI auto-detection from OPENROUTER_API_KEY env var
- Interactive provider selection in `repowise init`
- Embedder selection in `repowise serve`
- Server provider catalog for web UI
- No new pip dependency (uses existing openai package)

Tests:
- 13 unit tests (construction, generation, error mapping, headers)
- Registry test updated (builtin count 6 → 7)
- Integration test (skipped without OPENROUTER_API_KEY)
@RaghavChamadiya
Copy link
Copy Markdown
Collaborator

This is really thorough, probably the most complete provider PR we've gotten. LLM + embedder + CLI + server catalog all done, and the honest known limitations section is appreciated.

One thing to fix before merge:

  1. cost_tracker dead code (blocking): the constructor accepts cost_tracker but hardcodes it to None, and then there's an if self._cost_tracker block in generate() that can never execute. Either remove the parameter entirely or add a log warning that cost tracking is disabled for OpenRouter. Just don't want dead code sitting there confusing people.

Also flagging for awareness (not blocking):

  1. stream_chat duplication: the streaming logic is identical to OpenAI's, about 95 lines copy-pasted. Would be nice to extract a shared base eventually, but not blocking this PR on it.

  2. Tests: no coverage for stream_chat or OpenRouterEmbedder. Even basic construction + mock tests would help, but also not blocking.

Should be a quick fix, happy to merge after (1).

Remove the cost_tracker parameter and unreachable if-block from
generate(). OpenRouter proxies 200+ models with varying prices,
so cost tracking is documented as unsupported — users should check
the OpenRouter dashboard instead.
- stream_chat: text deltas, tool calls, rate limit error (3 tests)
- OpenRouterEmbedder: construction, dimensions, embedding, base URL (12 tests)
@oglenyaboss
Copy link
Copy Markdown
Author

Thanks for the review!

  1. cost_tracker dead code — fixed in c1c1773. Removed the cost_tracker parameter and the unreachable if self._cost_tracker block entirely. The constructor now accepts **_kwargs so it won't break if the pipeline passes cost_tracker= at instantiation time.

  2. stream_chat duplication — agreed, the shared base extraction makes sense. Happy to tackle that in a follow-up PR if you'd like.

  3. Tests — added in 57d40c7:

  • stream_chat: text deltas, tool calls, and rate limit error (3 tests)
  • OpenRouterEmbedder: construction, dimensions, embedding, and base URL verification (12 tests)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants