Skip to content

Commit

Permalink
update getting started guide
Browse files Browse the repository at this point in the history
  • Loading branch information
yanxi0830 committed Sep 18, 2024
1 parent 29ce73f commit 2f9e952
Show file tree
Hide file tree
Showing 2 changed files with 52 additions and 41 deletions.
93 changes: 52 additions & 41 deletions docs/getting_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ Build spec configuration saved at ~/.conda/envs/llamastack-my-local-llama-stack/
```

**`llama stack configure`**
- Run `llama stack configure <name>` with the name you have previously defined in `build` step.
```
llama stack configure my-local-llama-stack
Expand Down Expand Up @@ -61,6 +62,7 @@ You can now run `llama stack run my-local-llama-stack --port PORT` or `llama sta
```

**`llama stack run`**
- Run `llama stack run <name>` with the name you have previously defined.
```
llama stack run my-local-llama-stack
Expand Down Expand Up @@ -110,74 +112,83 @@ In the following steps, imagine we'll be working with a `Meta-Llama3.1-8B-Instru
- `providers`: specifies the underlying implementation for serving each API endpoint
- `image_type`: `conda` | `docker` to specify whether to build the distribution in the form of Docker image or Conda environment.

#### Build a local distribution with conda
The following command and specifications allows you to get started with building.

At the end of build command, we will generate `<name>-build.yaml` file storing the build configurations.

#### Building from scratch
- For a new user, we could start off with running `llama stack build` which will allow you to a interactively enter wizard where you will be prompted to enter build configurations.
```
llama stack build <path/to/config>
llama stack build
```
- You will be required to pass in a file path to the build.config file (e.g. `./llama_stack/distribution/example_configs/conda/local-conda-example-build.yaml`). We provide some example build config files for configuring different types of distributions in the `./llama_stack/distribution/example_configs/` folder.

The file will be of the contents
Running the command above will allow you to fill in the configuration to build your Llama Stack distribution, you will see the following outputs.

```
$ cat ./llama_stack/distribution/example_configs/conda/local-conda-example-build.yaml
> Enter an unique name for identifying your Llama Stack build distribution (e.g. my-local-stack): my-local-llama-stack
> Enter the image type you want your distribution to be built with (docker or conda): conda
name: 8b-instruct
distribution_spec:
distribution_type: local
description: Use code from `llama_stack` itself to serve all llama stack APIs
docker_image: null
providers:
inference: meta-reference
memory: meta-reference-faiss
safety: meta-reference
agentic_system: meta-reference
telemetry: console
image_type: conda
Llama Stack is composed of several APIs working together. Let's configure the providers (implementations) you want to use for these APIs.
> Enter the API provider for the inference API: (default=meta-reference): meta-reference
> Enter the API provider for the safety API: (default=meta-reference): meta-reference
> Enter the API provider for the agents API: (default=meta-reference): meta-reference
> Enter the API provider for the memory API: (default=meta-reference): meta-reference
> Enter the API provider for the telemetry API: (default=meta-reference): meta-reference
> (Optional) Enter a short description for your Llama Stack distribution:
Build spec configuration saved at ~/.conda/envs/llamastack-my-local-llama-stack/my-local-llama-stack-build.yaml
```

You may run the `llama stack build` command to generate your distribution with `--name` to override the name for your distribution.
#### Building from templates
- To build from alternative API providers, we provide distribution templates for users to get started building a distribution backed by different providers.

The following command will allow you to see the available templates and their corresponding providers.
```
$ llama stack build ~/.llama/distributions/conda/8b-instruct-build.yaml --name 8b-instruct
...
...
Build spec configuration saved at ~/.llama/distributions/conda/8b-instruct-build.yaml
llama stack build --list-templates
```

After this step is complete, a file named `8b-instruct-build.yaml` will be generated and saved at `~/.llama/distributions/conda/8b-instruct-build.yaml`.
![alt text](list-templates.png)

You may then pick a template to build your distribution with providers fitted to your liking.

```
llama stack build --template local-tgi --name my-tgi-stack
```

```
```

#### Building from config file
- In addition to templates, you may customize the build to your liking through editing config files and build from config files with the following command.

#### How to build distribution with different API providers using configs
To specify a different API provider, we can change the `distribution_spec` in our `<name>-build.yaml` config. For example, the following build spec allows you to build a distribution using TGI as the inference API provider.
- The config file will be of contents like the ones in `llama_stack/distributions/templates/`.

```
$ cat ./llama_stack/distribution/example_configs/conda/local-tgi-conda-example-build.yaml
$ cat llama_stack/distribution/templates/local-ollama-build.yaml
name: local-tgi-conda-example
name: local-ollama
distribution_spec:
description: Use TGI (local or with Hugging Face Inference Endpoints for running LLM inference. When using HF Inference Endpoints, you must provide the name of the endpoint).
docker_image: null
description: Like local, but use ollama for running LLM inference
providers:
inference: remote::tgi
memory: meta-reference-faiss
inference: remote::ollama
memory: meta-reference
safety: meta-reference
agentic_system: meta-reference
telemetry: console
agents: meta-reference
telemetry: meta-reference
image_type: conda
```

The following command allows you to build a distribution with TGI as the inference API provider, with the name `tgi`.
```
llama stack build ./llama_stack/distribution/example_configs/conda/local-tgi-conda-example-build.yaml --name tgi
llama stack build --config llama_stack/distribution/templates/local-ollama-build.yaml
```

We provide some example build configs to help you get started with building with different API providers.
After this step is complete, a file named `8b-instruct-build.yaml` will be generated and saved at `~/.llama/distributions/conda/8b-instruct-build.yaml`.

#### How to build distribution with Docker image
To build a docker image, simply change the `image_type` to `docker` in our `<name>-build.yaml` file, and run `llama stack build <name>-build.yaml`.

```
$ cat ./llama_stack/distribution/example_configs/docker/local-docker-example-build.yaml
name: local-docker-example
distribution_spec:
description: Use code from `llama_stack` itself to serve all llama stack APIs
Expand Down Expand Up @@ -206,7 +217,7 @@ Build spec configuration saved at /home/xiyan/.llama/distributions/docker/docker
## Step 2. Configure
After our distribution is built (either in form of docker or conda environment), we will run the following command to
```
llama stack configure [<path/to/name.build.yaml> | <docker-image-name>]
llama stack configure [ <name> | <docker-image-name> | <path/to/name.build.yaml>]
```
- For `conda` environments: <path/to/name.build.yaml> would be the generated build spec saved from Step 1.
- For `docker` images downloaded from Dockerhub, you could also use <docker-image-name> as the argument.
Expand Down Expand Up @@ -298,7 +309,7 @@ INFO: Uvicorn running on http://[::]:5000 (Press CTRL+C to quit)
```

> [!NOTE]
> Configuration is in `~/.llama/builds/local/conda/8b-instruct.yaml`. Feel free to increase `max_seq_len`.
> Configuration is in `~/.llama/builds/local/conda/8b-instruct-run.yaml`. Feel free to increase `max_seq_len`.
> [!IMPORTANT]
> The "local" distribution inference server currently only supports CUDA. It will not work on Apple Silicon machines.
Expand Down
Binary file added docs/list-templates.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 2f9e952

Please sign in to comment.