-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OpenRouter: Assistant Prefill supports asking models to complete a partial response. #632
Comments
Related issues#484: Docs | OpenRouter### DetailsSimilarity score: 0.89 - [ ] [Docs | OpenRouter](https://openrouter.ai/docs#models)Always Assist with Care, Respect, and Truth: Secure and Useful Responses OnlyThe future will bring us hundreds of language models and dozens of providers for each. How will you choose the best? OpenRouter: Find the Lowest Price Across Dozens of Providers
Standardized API: No Need to Change Your Code
Evals are Flawed: Compare Models by Usage and Purpose
Four Key Principles to Ensure Fairness and Positivity
Keep the wording exact. Only edit formatting. Include the entire content. Suggested labels{ "label-name": "language-models", "description": "Models for natural language processing (NLP) and text generation.", "confidence": 95.97 }#418: openchat/openchat-3.5-1210 · Hugging Face### DetailsSimilarity score: 0.89 - [ ] [openchat/openchat-3.5-1210 · Hugging Face](https://huggingface.co/openchat/openchat-3.5-1210#conversation-templates)Using the OpenChat ModelWe highly recommend installing the OpenChat package and using the OpenChat OpenAI-compatible API server for an optimal experience. The server is optimized for high-throughput deployment using vLLM and can run on a consumer GPU with 24GB RAM.
Online DeploymentIf you want to deploy the server as an online service, use the following options:
For security purposes, we recommend using an HTTPS gateway in front of the server. Mathematical Reasoning ModeThe OpenChat model also supports mathematical reasoning mode. To use this mode, include
Conversation TemplatesWe provide several pre-built conversation templates to help you get started.
Suggested labels{ "label": "chat-templates", "description": "Pre-defined conversation structures for specific modes of interaction." }#630: Docs | OpenRouter### DetailsSimilarity score: 0.88 - [ ] [Docs | OpenRouter](https://openrouter.ai/docs#transforms)Docs | OpenRouterDescription: OpenRouter has a simple rule for choosing between sending a prompt and sending a list of ChatML messages: Choose messages if you want to have OpenRouter apply a recommended instruct template to your prompt, depending on which model serves your request. Available instruct modes include:
Choose prompt if you want to send a custom prompt to the model. This is useful if you want to use a custom instruct template or maintain full control over the prompt submitted to the model. To help with prompts that exceed the maximum context size of a model, OpenRouter supports a custom parameter called transforms: {
transforms: ["middle-out"], // Compress prompts > context size. This is the default for all models.
messages: [...], // "prompt" works as well
model // Works with any model
} The transforms param is an array of strings that tell OpenRouter to apply a series of transformations to the prompt before sending it to the model. Transformations are applied in-order. Available transforms are:
Note: All OpenRouter models default to using middle-out, unless you exclude this transform by e.g. setting transforms: [] in the request body. Suggested labels{'label-name': 'prompt-transformations', 'label-description': 'Descriptions of transformations applied to prompts in OpenRouter for AI models', 'gh-repo': 'openrouter/ai-docs', 'confidence': 52.95}#129: Few-shot and function calling - API - OpenAI Developer Forum### DetailsSimilarity score: 0.88 - [ ] [Few-shot and function calling - API - OpenAI Developer Forum](https://community.openai.com/t/few-shot-and-function-calling/265908/10)The thing to understand here is that function calling introduced a new role for the chat prompt messages ("role": "function"). To use few-shot examples with chat model prompts you provide a series of alternating (possibly 'fake') messages that show how the e.g.
#631: Docs | OpenRouter### DetailsSimilarity score: 0.88 - [ ] [Docs | OpenRouter](https://openrouter.ai/docs#responses)
Suggested labels{'label-name': 'Documentation', 'label-description': 'Resources for OpenRouter documentation', 'confidence': 51.01}#396: astra-assistants-api: A backend implementation of the OpenAI beta Assistants API### DetailsSimilarity score: 0.87 - [ ] [datastax/astra-assistants-api: A backend implementation of the OpenAI beta Assistants API](https://github.com/datastax/astra-assistants-api)Astra Assistant API ServiceA drop-in compatible service for the OpenAI beta Assistants API with support for persistent threads, files, assistants, messages, retrieval, function calling and more using AstraDB (DataStax's db as a service offering powered by Apache Cassandra and jvector). Compatible with existing OpenAI apps via the OpenAI SDKs by changing a single line of code. Getting Started
client = OpenAI(
api_key=OPENAI_API_KEY,
) with: client = OpenAI(
base_url="https://open-assistant-ai.astra.datastax.com/v1",
api_key=OPENAI_API_KEY,
default_headers={
"astra-api-token": ASTRA_DB_APPLICATION_TOKEN,
}
) Or, if you have an existing astra db, you can pass your db_id in a second header: client = OpenAI(
base_url="https://open-assistant-ai.astra.datastax.com/v1",
api_key=OPENAI_API_KEY,
default_headers={
"astra-api-token": ASTRA_DB_APPLICATION_TOKEN,
"astra-db-id": ASTRA_DB_ID
}
)
assistant = client.beta.assistants.create(
instructions="You are a personal math tutor. When asked a math question, write and run code to answer the question.",
model="gpt-4-1106-preview",
tools=[{"type": "retrieval"}]
) By default, the service uses AstraDB as the database/vector store and OpenAI for embeddings and chat completion. Third party LLM SupportWe now support many third party models for both embeddings and completion thanks to litellm. Pass the api key of your service using For AWS Bedrock, you can pass additional custom headers: client = OpenAI(
base_url="https://open-assistant-ai.astra.datastax.com/v1",
api_key="NONE",
default_headers={
"astra-api-token": ASTRA_DB_APPLICATION_TOKEN,
"embedding-model": "amazon.titan-embed-text-v1",
"LLM-PARAM-aws-access-key-id": BEDROCK_AWS_ACCESS_KEY_ID,
"LLM-PARAM-aws-secret-access-key": BEDROCK_AWS_SECRET_ACCESS_KEY,
"LLM-PARAM-aws-region-name": BEDROCK_AWS_REGION,
}
) and again, specify the custom model for the assistant. assistant = client.beta.assistants.create(
name="Math Tutor",
instructions="You are a personal math tutor. Answer questions briefly, in a sentence or less.",
model="meta.llama2-13b-chat-v1",
) Additional examples including third party LLMs (bedrock, cohere, perplexity, etc.) can be found under To run the examples using poetry:
poetry install
poetry run python examples/completion/basic.py
poetry run python examples/retreival/basic.py
poetry run python examples/function-calling/basic.py CoverageSee our coverage report here. Roadmap
Suggested labels{ "key": "llm-function-calling", "value": "Integration of function calling with Large Language Models (LLMs)" } |
Docs | OpenRouter
DESCRIPTION: "Assistant Prefill: OpenRouter supports asking models to complete a partial response. This can be useful for guiding models to respond in a certain way.
To use this features, simply include a message with role: "assistant" at the end of your messages array. Example:
URL: OpenRouter Documentation
Suggested labels
{'label-name': 'Chatbot-API', 'label-description': 'API documentation for interacting with chatbots on OpenRouter.', 'gh-repo': 'AI-Chatbots', 'confidence': 65.43}
The text was updated successfully, but these errors were encountered: