diff --git a/docs/getting_started.md b/docs/getting_started.md index 0ab3461dd..c5bd75979 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -28,6 +28,7 @@ Build spec configuration saved at ~/.conda/envs/llamastack-my-local-llama-stack/ ``` **`llama stack configure`** +- Run `llama stack configure ` with the name you have previously defined in `build` step. ``` llama stack configure my-local-llama-stack @@ -61,6 +62,7 @@ You can now run `llama stack run my-local-llama-stack --port PORT` or `llama sta ``` **`llama stack run`** +- Run `llama stack run ` with the name you have previously defined. ``` llama stack run my-local-llama-stack @@ -110,74 +112,83 @@ In the following steps, imagine we'll be working with a `Meta-Llama3.1-8B-Instru - `providers`: specifies the underlying implementation for serving each API endpoint - `image_type`: `conda` | `docker` to specify whether to build the distribution in the form of Docker image or Conda environment. -#### Build a local distribution with conda -The following command and specifications allows you to get started with building. + +At the end of build command, we will generate `-build.yaml` file storing the build configurations. + +#### Building from scratch +- For a new user, we could start off with running `llama stack build` which will allow you to a interactively enter wizard where you will be prompted to enter build configurations. ``` -llama stack build +llama stack build ``` -- You will be required to pass in a file path to the build.config file (e.g. `./llama_stack/distribution/example_configs/conda/local-conda-example-build.yaml`). We provide some example build config files for configuring different types of distributions in the `./llama_stack/distribution/example_configs/` folder. -The file will be of the contents +Running the command above will allow you to fill in the configuration to build your Llama Stack distribution, you will see the following outputs. + ``` -$ cat ./llama_stack/distribution/example_configs/conda/local-conda-example-build.yaml +> Enter an unique name for identifying your Llama Stack build distribution (e.g. my-local-stack): my-local-llama-stack +> Enter the image type you want your distribution to be built with (docker or conda): conda -name: 8b-instruct -distribution_spec: - distribution_type: local - description: Use code from `llama_stack` itself to serve all llama stack APIs - docker_image: null - providers: - inference: meta-reference - memory: meta-reference-faiss - safety: meta-reference - agentic_system: meta-reference - telemetry: console -image_type: conda + Llama Stack is composed of several APIs working together. Let's configure the providers (implementations) you want to use for these APIs. +> Enter the API provider for the inference API: (default=meta-reference): meta-reference +> Enter the API provider for the safety API: (default=meta-reference): meta-reference +> Enter the API provider for the agents API: (default=meta-reference): meta-reference +> Enter the API provider for the memory API: (default=meta-reference): meta-reference +> Enter the API provider for the telemetry API: (default=meta-reference): meta-reference + + > (Optional) Enter a short description for your Llama Stack distribution: + +Build spec configuration saved at ~/.conda/envs/llamastack-my-local-llama-stack/my-local-llama-stack-build.yaml ``` -You may run the `llama stack build` command to generate your distribution with `--name` to override the name for your distribution. +#### Building from templates +- To build from alternative API providers, we provide distribution templates for users to get started building a distribution backed by different providers. + +The following command will allow you to see the available templates and their corresponding providers. ``` -$ llama stack build ~/.llama/distributions/conda/8b-instruct-build.yaml --name 8b-instruct -... -... -Build spec configuration saved at ~/.llama/distributions/conda/8b-instruct-build.yaml +llama stack build --list-templates ``` -After this step is complete, a file named `8b-instruct-build.yaml` will be generated and saved at `~/.llama/distributions/conda/8b-instruct-build.yaml`. +![alt text](list-templates.png) + +You may then pick a template to build your distribution with providers fitted to your liking. + +``` +llama stack build --template local-tgi --name my-tgi-stack +``` + +``` +``` + +#### Building from config file +- In addition to templates, you may customize the build to your liking through editing config files and build from config files with the following command. -#### How to build distribution with different API providers using configs -To specify a different API provider, we can change the `distribution_spec` in our `-build.yaml` config. For example, the following build spec allows you to build a distribution using TGI as the inference API provider. +- The config file will be of contents like the ones in `llama_stack/distributions/templates/`. ``` -$ cat ./llama_stack/distribution/example_configs/conda/local-tgi-conda-example-build.yaml +$ cat llama_stack/distribution/templates/local-ollama-build.yaml -name: local-tgi-conda-example +name: local-ollama distribution_spec: - description: Use TGI (local or with Hugging Face Inference Endpoints for running LLM inference. When using HF Inference Endpoints, you must provide the name of the endpoint). - docker_image: null + description: Like local, but use ollama for running LLM inference providers: - inference: remote::tgi - memory: meta-reference-faiss + inference: remote::ollama + memory: meta-reference safety: meta-reference - agentic_system: meta-reference - telemetry: console + agents: meta-reference + telemetry: meta-reference image_type: conda ``` -The following command allows you to build a distribution with TGI as the inference API provider, with the name `tgi`. ``` -llama stack build ./llama_stack/distribution/example_configs/conda/local-tgi-conda-example-build.yaml --name tgi +llama stack build --config llama_stack/distribution/templates/local-ollama-build.yaml ``` -We provide some example build configs to help you get started with building with different API providers. +After this step is complete, a file named `8b-instruct-build.yaml` will be generated and saved at `~/.llama/distributions/conda/8b-instruct-build.yaml`. #### How to build distribution with Docker image To build a docker image, simply change the `image_type` to `docker` in our `-build.yaml` file, and run `llama stack build -build.yaml`. ``` -$ cat ./llama_stack/distribution/example_configs/docker/local-docker-example-build.yaml - name: local-docker-example distribution_spec: description: Use code from `llama_stack` itself to serve all llama stack APIs @@ -206,7 +217,7 @@ Build spec configuration saved at /home/xiyan/.llama/distributions/docker/docker ## Step 2. Configure After our distribution is built (either in form of docker or conda environment), we will run the following command to ``` -llama stack configure [ | ] +llama stack configure [ | | ] ``` - For `conda` environments: would be the generated build spec saved from Step 1. - For `docker` images downloaded from Dockerhub, you could also use as the argument. @@ -298,7 +309,7 @@ INFO: Uvicorn running on http://[::]:5000 (Press CTRL+C to quit) ``` > [!NOTE] -> Configuration is in `~/.llama/builds/local/conda/8b-instruct.yaml`. Feel free to increase `max_seq_len`. +> Configuration is in `~/.llama/builds/local/conda/8b-instruct-run.yaml`. Feel free to increase `max_seq_len`. > [!IMPORTANT] > The "local" distribution inference server currently only supports CUDA. It will not work on Apple Silicon machines. diff --git a/docs/list-templates.png b/docs/list-templates.png new file mode 100644 index 000000000..5b17641ef Binary files /dev/null and b/docs/list-templates.png differ