Skip to content

DMR: GUI docs #22685

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

DMR: GUI docs #22685

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion content/manuals/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ params:
- title: Docker Model Runner
description: View and manage your local models.
icon: view_in_ar
link: /model-runner/
link: /ai/model-runner/
- title: MCP Catalog and Toolkit
description: Augment your AI workflow with MCP servers.
icon: /assets/icons/toolbox.svg
Expand Down
229 changes: 45 additions & 184 deletions content/manuals/ai/model-runner.md → content/manuals/ai/model-runner/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,27 +8,30 @@
group: AI
weight: 20
description: Learn how to use Docker Model Runner to manage and run AI models.
keywords: Docker, ai, model runner, docker deskotp, llm
keywords: Docker, ai, model runner, docker desktop, llm

Check failure on line 11 in content/manuals/ai/model-runner/_index.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[Vale.Terms] Use 'Docker Desktop' instead of 'docker desktop'.

Raw Output:
{"message": "[Vale.Terms] Use 'Docker Desktop' instead of 'docker desktop'.", "location": {"path": "content/manuals/ai/model-runner/_index.md", "range": {"start": {"line": 11, "column": 37}}}, "severity": "ERROR"}

Check failure on line 11 in content/manuals/ai/model-runner/_index.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[Vale.Spelling] Did you really mean 'llm'?

Raw Output:
{"message": "[Vale.Spelling] Did you really mean 'llm'?", "location": {"path": "content/manuals/ai/model-runner/_index.md", "range": {"start": {"line": 11, "column": 53}}}, "severity": "ERROR"}
aliases:
- /desktop/features/model-runner/
- /ai/model-runner/
- /model-runner/
---

{{< summary-bar feature_name="Docker Model Runner" >}}

The Docker Model Runner plugin lets you:
## Key features

- [Pull models from Docker Hub](https://hub.docker.com/u/ai)
- Run AI models directly from the command line
- Manage local models (add, list, remove)
- Interact with models using a submitted prompt or in chat mode in the CLI or Docker Desktop Dashboard
- Push models to Docker Hub
- [Pull and push models to and from Docker Hub](https://hub.docker.com/u/ai)
- Run and interact with AI models directly from the command line or from Docker Desktop
- Manage local models and display logs

## How it works

Models are pulled from Docker Hub the first time they're used and stored locally. They're loaded into memory only at runtime when a request is made, and unloaded when not in use to optimize resources. Since models can be large, the initial pull may take some time — but after that, they're cached locally for faster access. You can interact with the model using [OpenAI-compatible APIs](#what-api-endpoints-are-available).

> [!TIP]
>
> Using Testcontainers or Docker Compose? [Testcontainers for Java](https://java.testcontainers.org/modules/docker_model_runner/) and [Go](https://golang.testcontainers.org/modules/dockermodelrunner/), and [Docker Compose](/manuals/compose/how-tos/model-runner.md) now support Docker Model Runner.
> Using Testcontainers or Docker Compose?
> [Testcontainers for Java](https://java.testcontainers.org/modules/docker_model_runner/)
> and [Go](https://golang.testcontainers.org/modules/dockermodelrunner/), and
> [Docker Compose](/manuals/compose/how-tos/model-runner.md) now support Docker Model Runner.

## Enable Docker Model Runner

Expand All @@ -42,192 +45,60 @@

You can now use the `docker model` command in the CLI and view and interact with your local models in the **Models** tab in the Docker Desktop Dashboard.

## Available commands

### Model runner status

Check whether the Docker Model Runner is active and displays the current inference engine:

```console
$ docker model status
```

### View all commands

Displays help information and a list of available subcommands.

```console
$ docker model help
```

Output:

```text
Usage: docker model COMMAND

Commands:
list List models available locally
pull Download a model from Docker Hub
rm Remove a downloaded model
run Run a model interactively or with a prompt
status Check if the model runner is running
version Show the current version
```

### Pull a model

Pulls a model from Docker Hub to your local environment.

```console
$ docker model pull <model>
```

Example:

```console
$ docker model pull ai/smollm2
```

Output:

```text
Downloaded: 257.71 MB
Model ai/smollm2 pulled successfully
```

The models also display in the Docker Desktop Dashboard.

#### Pull from Hugging Face

You can also pull GGUF models directly from [Hugging Face](https://huggingface.co/models?library=gguf).

```console
$ docker model pull hf.co/<model-you-want-to-pull>
```

For example:

```console
$ docker model pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF
```

Pulls the [bartowski/Llama-3.2-1B-Instruct-GGUF](https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF).

### List available models

Lists all models currently pulled to your local environment.

```console
$ docker model list
```

You will see something similar to:
## Install a model

```text
+MODEL PARAMETERS QUANTIZATION ARCHITECTURE MODEL ID CREATED SIZE
+ai/smollm2 361.82 M IQ2_XXS/Q4_K_M llama 354bf30d0aa3 3 days ago 256.35 MiB
```
Models are installed locally.

### Run a model
{{< tabs >}}
{{< tab name="From Docker Desktop">}}

Run a model and interact with it using a submitted prompt or in chat mode. When you run a model, Docker
calls an Inference Server API endpoint hosted by the Model Runner through Docker Desktop. The model
stays in memory until another model is requested, or until a pre-defined inactivity timeout is reached (currently 5 minutes).
1. Select **Models** and select the **Docker Hub** tab.
2. Find the model of your choice and select **Pull**.

You do not have to use `Docker model run` before interacting with a specific model from a
host process or from within a container. Model Runner transparently loads the requested model on-demand, assuming it has been
pulled beforehand and is locally available.
{{< /tab >}}
{{< tab name="From the Docker CLI">}}

#### One-time prompt
Use the [`docker model pull` command](/reference/cli/docker/).

```console
$ docker model run ai/smollm2 "Hi"
```
{{< /tab >}}
{{< /tabs >}}

Output:
## Run a model

```text
Hello! How can I assist you today?
```

#### Interactive chat

```console
$ docker model run ai/smollm2
```
Models are installed locally.

Output:
{{< tabs >}}
{{< tab name="From Docker Desktop">}}

```text
Interactive chat mode started. Type '/bye' to exit.
> Hi
Hi there! It's SmolLM, AI assistant. How can I help you today?
> /bye
Chat session ended.
```
Select **Models** and select the **Local** tab and click the play button.

Check warning on line 73 in content/manuals/ai/model-runner/_index.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[Docker.RecommendedWords] Consider using 'select' instead of 'click'

Raw Output:
{"message": "[Docker.RecommendedWords] Consider using 'select' instead of 'click'", "location": {"path": "content/manuals/ai/model-runner/_index.md", "range": {"start": {"line": 73, "column": 52}}}, "severity": "INFO"}
The interactive chat screen opens.

> [!TIP]
>
> You can also use chat mode in the Docker Desktop Dashboard when you select the model in the **Models** tab.
{{< /tab >}}
{{< tab name="From the Docker CLI">}}

### Push a model to Docker Hub
Use the [`docker model run` command](/reference/cli/docker/).

To push your model to Docker Hub:
{{< /tab >}}
{{< /tabs >}}

```console
$ docker model push <namespace>/<model>
```
## Troubleshooting

### Tag a model
To troubleshoot potential issues, display the logs:

To specify a particular version or variant of the model:
{{< tabs >}}
{{< tab name="From Docker Desktop">}}

```console
$ docker model tag
```
Select **Models** and select the **Logs** tab.

If no tag is provided, Docker defaults to `latest`.
{{< /tab >}}
{{< tab name="From the Docker CLI">}}

### View the logs
Use the [`docker model log` command](/reference/cli/docker/).

Fetch logs from Docker Model Runner to monitor activity or debug issues.
{{< /tab >}}
{{< /tabs >}}

```console
$ docker model logs
```

The following flags are accepted:

- `-f`/`--follow`: View logs with real-time streaming
- `--no-engines`: Exclude inference engine logs from the output

### Remove a model

Removes a downloaded model from your system.

```console
$ docker model rm <model>
```

Output:

```text
Model <model> removed successfully
```

### Package a model

Packages a GGUF file into a Docker model OCI artifact, with optional licenses, and pushes it to the specified registry.

```console
$ docker model package \
--gguf ./model.gguf \
--licenses license1.txt \
--licenses license2.txt \
--push registry.example.com/ai/custom-model
```

## Integrate the Docker Model Runner into your software development lifecycle
## Example: Integrate Docker Model Runner into your software development lifecycle

Check warning on line 101 in content/manuals/ai/model-runner/_index.md

View workflow job for this annotation

GitHub Actions / vale 

[vale] reported by reviewdog 🐶
[Docker.HeadingLength] Try to keep headings short (< 8 words).

Raw Output:
{"message": "[Docker.HeadingLength] Try to keep headings short (\u003c 8 words).", "location": {"path": "content/manuals/ai/model-runner/_index.md", "range": {"start": {"line": 101, "column": 4}}}, "severity": "INFO"}

You can now start building your Generative AI application powered by the Docker Model Runner.

Expand Down Expand Up @@ -287,7 +158,6 @@
> [!NOTE]
> You can omit `llama.cpp` from the path. For example: `POST /engines/v1/chat/completions`.


### How do I interact through the OpenAI API?

#### From within a container
Expand Down Expand Up @@ -399,12 +269,3 @@
## Share feedback

Thanks for trying out Docker Model Runner. Give feedback or report any bugs you may find through the **Give feedback** link next to the **Enable Docker Model Runner** setting.

## Disable the feature

To disable Docker Model Runner:

1. Open the **Settings** view in Docker Desktop.
2. Navigate to the **Beta** tab in **Features in development**.
3. Clear the **Enable Docker Model Runner** checkbox.
4. Select **Apply & restart**.