Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

Benchmarking : Misc updates #95

Merged
merged 6 commits into from
Mar 11, 2024
Merged

Benchmarking : Misc updates #95

merged 6 commits into from
Mar 11, 2024

Conversation

varun-sundar-rabindranath

SUMMARY:
Fixes and Quality-of-life changes

  • Fix the vllm engine temperature to 0.0 so the text generation is deterministic
  • Fix time-per-output-token metric computation
  • Add num_warmup_prompts and log_model_io options to benchmark throughput

TEST PLAN:
Manual testing

@mgoin mgoin merged commit aebf20b into main Mar 11, 2024
2 checks passed
@mgoin mgoin deleted the varun/benchmark-misc-updates branch March 11, 2024 20:20
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants