Skip to content

Commit 467a8ee

Browse files
cjrolomingshl
cjrolo
and
mingshl
authored
Ollama connector blueprint (#4160)
* Ollama connector blueprint Signed-off-by: Carlos Rolo <carlos.rolo@netapp.com> * Improved doc and added a missing step Signed-off-by: Carlos Rolo <carlos.rolo@netapp.com> * Blueprint finished, working on tutorial --------- Signed-off-by: Carlos Rolo <carlos.rolo@netapp.com> Co-authored-by: Mingshi Liu <mingshl@amazon.com>
1 parent 02b217e commit 467a8ee

File tree

2 files changed

+603
-0
lines changed

2 files changed

+603
-0
lines changed

‎docs/remote_inference_blueprints/ollama_connector_chat_blueprint.md‎

Lines changed: 229 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,229 @@
1+
# Ollama (OpenAI compatible) connector blueprint example for chat
2+
3+
This is an AI connector blueprint for Ollama or any other local/self-hosted LLM as long as it is OpenAI compatible (Ollama, llama.cpp, vLLM, etc)
4+
5+
## 1. Add connector endpoint to trusted URLs
6+
7+
Adjust the Regex to your local IP. The following example allows all URLs.
8+
9+
```json
10+
PUT /_cluster/settings
11+
{
12+
"persistent": {
13+
"plugins.ml_commons.trusted_connector_endpoints_regex": [
14+
".*$"
15+
]
16+
}
17+
}
18+
```
19+
20+
## 2. Enable private addresses
21+
22+
```json
23+
PUT /_cluster/settings
24+
{
25+
"persistent": {
26+
"plugins.ml_commons.connector.private_ip_enabled": true
27+
}
28+
}
29+
```
30+
31+
## 3. Create the connector
32+
33+
In a local setting, `openAI_key` might not be needed. In case you can either set it to something irrelevant, or if removed, you need to update the `Authorization` header in the `actions`.
34+
35+
```json
36+
POST /_plugins/_ml/connectors/_create
37+
{
38+
"name": "<YOUR CONNECTOR NAME>",
39+
"description": "<YOUR CONNECTOR DESCRIPTION>",
40+
"version": "<YOUR CONNECTOR VERSION>",
41+
"protocol": "http",
42+
"parameters": {
43+
"endpoint": "127.0.0.1:11434",
44+
"model": "qwen3:4b"
45+
},
46+
"credential": {
47+
"openAI_key": "<YOUR API KEY HERE IF NEEDED>"
48+
},
49+
"actions": [
50+
{
51+
"action_type": "predict",
52+
"method": "POST",
53+
"url": "https://${parameters.endpoint}/v1/chat/completions",
54+
"headers": {
55+
"Authorization": "Bearer ${credential.openAI_key}"
56+
},
57+
"request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages} }"
58+
}
59+
]
60+
}
61+
```
62+
63+
### Sample response
64+
65+
```json
66+
{
67+
"connector_id": "Keq5FpkB72uHgF272LWj"
68+
}
69+
```
70+
71+
## 4. Register the model and deploy the model
72+
73+
Getting a model to work is a 2-step process. Register and Deploy.
74+
75+
### A: Register and Deploy in two steps
76+
77+
One way to do this is a `_register` call followed by a `_deploy` call.
78+
First you register:
79+
80+
```json
81+
POST /_plugins/_ml/models/_register
82+
{
83+
"name": "Local LLM Model",
84+
"function_name": "remote",
85+
"description": "Ollama model",
86+
"connector_id": "Keq5FpkB72uHgF272LWj"
87+
}
88+
```
89+
90+
You get a response like this:
91+
92+
```json
93+
{
94+
"task_id": "oEdPqZQBQwAL8-GOCJbw",
95+
"status": "CREATED",
96+
"model_id": "oUdPqZQBQwAL8-GOCZYL"
97+
}
98+
```
99+
100+
Take note of the `model_id`, it is needed for the `_deploy` call.
101+
102+
Then you deploy:
103+
104+
Use `model_id` in place of the `<MODEL_ID>` placeholder.
105+
106+
```json
107+
POST /_plugins/_ml/models/<MODEL_ID>/_deploy
108+
```
109+
110+
#### Sample response
111+
112+
Once you get a response like this, your model is ready to use.
113+
114+
```json
115+
{
116+
"task_id": "oEdPqZQBQwAL8-GOCJbw",
117+
"status": "CREATED",
118+
"model_id": "oUdPqZQBQwAL8-GOCZYL"
119+
}
120+
```
121+
122+
### B: Register and Deploy in a single step
123+
124+
Another way is doing the 2 steps at once with `deploy=true`:
125+
126+
```json
127+
POST /_plugins/_ml/models/_register?deploy=true
128+
{
129+
"name": "Local LLM Model",
130+
"function_name": "remote",
131+
"description": "Ollama model",
132+
"connector_id": "Keq5FpkB72uHgF272LWj"
133+
}
134+
```
135+
136+
#### Sample response
137+
138+
Once you get a response like this, your model is ready to use.
139+
140+
```json
141+
{
142+
"task_id": "oEdPqZQBQwAL8-GOCJbw",
143+
"status": "CREATED",
144+
"model_id": "oUdPqZQBQwAL8-GOCZYL"
145+
}
146+
```
147+
148+
### 5. Corresponding Predict request example
149+
150+
Notice how you have to create the whole message structure, not just the message to send.
151+
Use `model_id` in place of the `<MODEL_ID>` placeholder.
152+
153+
```json
154+
POST /_plugins/_ml/models/<MODEL_ID>/_predict
155+
{
156+
"parameters": {
157+
"messages": [
158+
{
159+
"role": "system",
160+
"content": "You are a helpful assistant."
161+
},
162+
{
163+
"role": "user",
164+
"content": "Why is the sky blue"
165+
}
166+
]
167+
}
168+
}
169+
```
170+
171+
### Sample response
172+
173+
```json
174+
{
175+
"inference_results": [
176+
{
177+
"output": [
178+
{
179+
"name": "response",
180+
"dataAsMap": {
181+
"choices": [
182+
{
183+
"finish_reason": "stop",
184+
"index": 0,
185+
"message": {
186+
"role": "assistant",
187+
"content": """The sky appears blue due to a phenomenon called Rayleigh scattering. Here's a simple explanation:
188+
189+
1. **Sunlight Composition**: Sunlight appears white, but it's actually a mix of all colors of the visible spectrum (red, orange, yellow, green, blue, indigo, violet).
190+
191+
2. **Atmospheric Scattering**: When sunlight enters Earth's atmosphere, it interacts with the gas molecules and tiny particles in the air. Shorter wavelengths of light (like blue and violet) are scattered more than other colors because they travel in shorter, smaller waves.
192+
193+
3. **Why Blue Dominates**: Although violet light is scattered even more than blue light, the sky appears blue, not violet, because:
194+
- Our eyes are more sensitive to blue light than violet light.
195+
- The sun emits more blue light than violet light.
196+
- Some of the violet light gets absorbed by the upper atmosphere.
197+
198+
4. **Time of Day**: The sky appears blue during the day because we're seeing the scattered blue light from all directions. At sunrise or sunset, the light has to pass through more of the atmosphere, scattering the blue light away and leaving mostly red and orange hues.
199+
200+
This scattering effect is named after Lord Rayleigh, who mathematically described the phenomenon in the 19th century."""
201+
}
202+
}
203+
],
204+
"created": 1757369906,
205+
"model": "qwen3:4b",
206+
"system_fingerprint": "b6259-cebb30fb",
207+
"object": "chat.completion",
208+
"usage": {
209+
"completion_tokens": 264,
210+
"prompt_tokens": 563,
211+
"total_tokens": 827
212+
},
213+
"id": "chatcmpl-iHioFpaxa8K2SXgAHd4FhQnbewLQ9PjB",
214+
"timings": {
215+
"prompt_n": 563,
216+
"prompt_ms": 293.518,
217+
"prompt_per_token_ms": 0.5213463587921847,
218+
"prompt_per_second": 1918.1106439809487,
219+
"predicted_n": 264,
220+
"predicted_ms": 5084.336,
221+
"predicted_per_token_ms": 19.258848484848485,
222+
"predicted_per_second": 51.92418439693993
223+
}
224+
}
225+
}
226+
],
227+
"status_code": 200
228+
}
229+
```

0 commit comments

Comments
 (0)