server : better security control for public deployments #9776
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ggerganov
approved these changes
Oct 8, 2024
dsx1986
pushed a commit
to dsx1986/llama.cpp
that referenced
this pull request
Oct 29, 2024
* server : more explicit endpoint access settings * protect /props endpoint * fix tests * update server docs * fix typo * fix tests
arthw
pushed a commit
to arthw/llama.cpp
that referenced
this pull request
Nov 15, 2024
* server : more explicit endpoint access settings * protect /props endpoint * fix tests * update server docs * fix typo * fix tests
arthw
pushed a commit
to arthw/llama.cpp
that referenced
this pull request
Nov 18, 2024
* server : more explicit endpoint access settings * protect /props endpoint * fix tests * update server docs * fix typo * fix tests
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Server REST API breaking changes
/slotsendpoint is now disabled by default, start server with--slotsto enable it/slotsand/props) requires a correct API key to access.Note: Only
/healthand/modelsare always publicly accessible"system_prompt"is removed from/completionsendpoint. It is now moved to POST/props(see documentation)Please note that GET
/propsis always enabled to avoid breaking the web UI.Why?
At the beginning, many functionalities of the server are exposed by default to make the development / testing process easier. It is supposed to be used locally, or in a private network (such as docker).
However, as more and more people want to use llama.cpp server in a production environment, letting everything exposed by default is quite risky, specially when most users only know and use OAI-compat endpoints.
The solution in the PR is to disable "advanced" functionalities by default, and let the users choice to enable them when they want. For example,
/slotsis now disabled by default, user must explicitly add--slotsargument to enable it.