Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

fixes issue identified by derek #83

Merged
merged 2 commits into from
Feb 29, 2024
Merged

fixes issue identified by derek #83

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5e23483
fixes issue identified by derek
robertgshaw2-neuralmagic Feb 29, 2024
eb780c1
added aio http comment
robertgshaw2-neuralmagic Feb 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion examples/production_monitoring/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,14 @@ Launch Prometheus and Grafana servers with `docker compose`:
docker compose up
```

Submit some sample requests to the server:
Submit some sample requests to the server (after `pip install aiohttp`):
```bash
wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

python3 ../../benchmarks/benchmark_serving.py \
--model mistralai/Mistral-7B-v0.1 \
--tokenizer mistralai/Mistral-7B-v0.1 \
--backend openai \
--endpoint /v1/completions \
--dataset ShareGPT_V3_unfiltered_cleaned_split.json \
--request-rate 3.0
Expand Down
Loading