server bench: fix bench not waiting for model load #7284
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
phymbert
approved these changes
May 14, 2024
mofosyne
added
examples
Review Complexity : Low
|
It looks like this change causes the server Benchmark that we run on the self-hosted runner to fail like this: https://github.com/ggerganov/llama.cpp/actions/runs/9094073377/job/24998422481 I tried to revert it and now the benchmark passes: https://github.com/ggerganov/llama.cpp/actions/runs/9112533114 I'm not sure why it is causing the error - any ideas how to fix? |
|
Yes, the problem is here: llama.cpp/examples/server/bench/bench.py Line 113 in 9afdffe It considers prometheus not started, which is not working as expected. Probably easier to revert and separate in another PR prometheus check vs llama.cpp server checks ? |
phymbert
added a commit
that referenced
this pull request
May 16, 2024
teleprint-me
pushed a commit
to teleprint-me/llama.cpp
that referenced
this pull request
May 17, 2024
teleprint-me
pushed a commit
to teleprint-me/llama.cpp
that referenced
this pull request
May 17, 2024
…#7284)" (ggerganov#7334) This reverts commit 583fd6b.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While working on #6828 I noticed that when using a large static n-ngam cache the benchmark would report 0 iterations for the first 8 minutes and then 30 iterations for the last 2 minutes. What seems to be happening is that
bench.pydoesn't correctly wait for the server to be ready so the clock starts ticking even while the n-gram cache is still being loaded. From what I can tell loading the model from disk can have the same issue if it's e.g. on an HDD.This PR makes it so that
bench.pywaits for response 200 (SERVER_STATE_READY) from the health endpoint for checking whether the server is actually ready. I'm not sure if there is a better way to implement this than what I did; I'm definitely open to suggestions.