[docs] add configuration doc to deployment guide #1578
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds an updated configuration doc to the deployment guide. It is intended to be a comprehensive configuration guide in the context of LMI.
It walks through the important configurations, what they mean, and how to set them.
It also covers the same configurations we had in our previous doc, but in a list form rather than table form. The table form is harder to parse compared to the list form in my opinion, but let me know if you disagree.
It also covers the environment variable translation at the end. Most of the focus is on the
serving.properties
format.This is intended to reflect the state of V8. I know we are working on many improvements to minimized the required set of configs, but I am not confident in recommending auto currently because it can easily fall back to the slow huggingface accelerate backend, and it cannot be used to select vllm.
For the
HF_MODEL_ID
experience, I think that is better served in a separate doc. This doc is intended for advanced users that want a comprehensive overview.