Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] add configuration doc to deployment guide #1578

Merged
merged 2 commits into from
Mar 1, 2024

Conversation

siddvenk
Copy link
Contributor

This PR adds an updated configuration doc to the deployment guide. It is intended to be a comprehensive configuration guide in the context of LMI.

It walks through the important configurations, what they mean, and how to set them.

It also covers the same configurations we had in our previous doc, but in a list form rather than table form. The table form is harder to parse compared to the list form in my opinion, but let me know if you disagree.

It also covers the environment variable translation at the end. Most of the focus is on the serving.properties format.


This is intended to reflect the state of V8. I know we are working on many improvements to minimized the required set of configs, but I am not confident in recommending auto currently because it can easily fall back to the slow huggingface accelerate backend, and it cannot be used to select vllm.

For the HF_MODEL_ID experience, I think that is better served in a separate doc. This doc is intended for advanced users that want a comprehensive overview.

@siddvenk siddvenk requested review from zachgk, frankfliu and a team as code owners February 28, 2024 16:43
@siddvenk siddvenk force-pushed the dg-configs branch 2 times, most recently from dbcee50 to b077aa0 Compare February 28, 2024 23:17
@lanking520 lanking520 merged commit 50193b1 into deepjavalibrary:master Mar 1, 2024
2 checks passed
@siddvenk siddvenk deleted the dg-configs branch April 26, 2024 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants