Description
🚀 The feature, motivation and pitch
An inference service that deploys models as serverless functions is critically dependent on container startup times to ensure it is responsive to changing traffic patterns. This is difficult to manage when model tensors stored in a container image bloat its size by tens of gigabytes. vLLM can now leverage tensorizer for its users in this PR.
With CoreWeave's tensorizer, model tensors can be loaded off HTTP/HTTPS, Redis, or S3 endpoints. By not embedding the model in the container image, users can reduce the container image size and the time it takes to load the model. This allows for fast, and therefore responsive, autoscaling.
tensorizer serializes and can additionally encrypt model tensors to be loaded and/or decrypted extremely fast during container startup. Decoupling the model from the container image also additionally allows users to be able to update models without having to rebuild the container images, saving time deploying new versions without waiting for container image builds or for container image cache to be populated.
Alternatives
No response
Additional context
