fix: Model Cache feature is under Generative AI instead of Predictive AI#571
fix: Model Cache feature is under Generative AI instead of Predictive AI#571Billy99 wants to merge 1 commit intokserve:mainfrom
Conversation
The Local Model Cache feature is only implemented for the Predictive AI use case, but in the docs on the website, the feature is documented under the Generative AI headings. This PR leaves current feature description as is, just moves the documentation links from Generative to Predictive. Signed-off-by: Billy McFall <22157057+Billy99@users.noreply.github.com>
✅ Deploy Preview for elastic-nobel-0aef7a ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Billy99
force-pushed
the
billy99-model-cache-to-pred
branch
2 times, most recently
from
7c7b53f to
ff67955
Compare
|
|
||
| ### LocalModel & LocalModelNode | ||
| Enables local model caching and management: | ||
| - **[Concepts](../../model-serving/generative-inference/modelcache/localmodel.md)**: Overview of local model caching in KServe. |
There was a problem hiding this comment.
I think ModelCache works for both scenarios; however, it is more useful for generative AI, where the models can get very big, no?
There was a problem hiding this comment.
When you read the feature description, it only describes InferenceService CRD, which is Predictive AI.
https://kserve.github.io/website/docs/model-serving/generative-inference/modelcache/localmodel
I recall in a previous discussion that it may work longer term with the LLMInferenceService CRD, but not there yet.
There was a problem hiding this comment.
You can run generative with the InferenceService type; the LLMINferenceService is designed for deployments that use LLM-D.
There was a problem hiding this comment.
Ok, let me reread some of the docs with that perspective.
There was a problem hiding this comment.
From Evolution: Dual-Track Strategy:
Strategic Separation:
- InferenceService: Remains the standard for Predictive AI (classification, regression, recommendations)
- LLMInferenceService: Dedicated to Generative AI with specialized optimizations
- Can you use InferenceService for LLMs? Yes, but only for basic single-node deployments. Advanced features like prefill-decode separation, multi-node orchestration, and intelligent scheduling are not available
So that's clear that InferenceService can be used for both Predictive AI and Generative AI, so I misunderstood. However, it makes it sound like "You can use for Generative AI but why would you ever because you lose all these features?".
And the QuickStart makes it sound like it's one or the other, which is probably where I got my original thought that InferenceService was only for Predictive AI.
Welcome to the KServe Quickstart Guide! This guide will help you set up a KServe Quickstart environment for testing and experimentation. KServe provides two deployment paths based on your use case:
- Generative AI (LLMInferenceService): For Large Language Models and generative AI workload
- Predictive AI (InferenceService): For traditional ML models and predictive inference workloads
So the description for this PR is wrong, Local Model Cache can be used for both Generative AI and Predictive AI. Totally fine closing this PR. But is it true that it is probably used more for Predictive AI? If so, does it make sense to move the Local Cache description to Predictive AI? If that's not the case, to remove confusion, maybe it should be moved up a level so it's common or under Model Storage, instead of tucked under Generative AI? Once again, happy to close this PR off, just brainstorming.
2 participants
The Local Model Cache feature is only implemented for the Predictive AI use case, but in the docs on the website, the feature is documented under the Generative AI headings. This PR leaves current feature description as is, just moves the documentation links from Generative to Predictive.