Open
Description
Description:
Right now, GuideLLM only works w/ GuideLLM through the OpenAI compatible API backend to take the target URI and perform benchmarking on a vLLM server. We want to also add support for the python vLLM backend so that we can create scripts for benchmarks w/ vLLM and use it in-line with Python.
Acceptance Criteria:
- Add support for the Python vLLM Backend
- Potentially helpful: https://github.com/triton-inference-server/vllm_backend - maybe not
Metadata
Metadata
Assignees
Type
Projects
Status
No status