Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

remove low-workload benchmarks that are flaky #156

Merged
merged 1 commit into from
Apr 1, 2024

Conversation

varun-sundar-rabindranath

Summary:
This PR removes the following benchmarking runs,

  • Sparse 2:4 : prompts 300, qps 1
  • Sparse 50 % : prompts 150, qps 0.5

Nightlies that reported flakiness / regression:

On the nightlies we trigger a perf. regression alert when the value change is more than 10%. For the cases being removed, the variance is in the range of [10% - 20%]

Test:
None

@varun-sundar-rabindranath
Copy link
Author

median_ttft_ms (the flaky metric) graph of the cases being removed,

sparse-2-4
sparse-50

@mgoin mgoin merged commit cbe584e into main Apr 1, 2024
2 checks passed
@mgoin mgoin deleted the varun/prune-flaky-benchmarks branch April 1, 2024 20:40
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants