Project Roadmap

[tgaddair]

WIP project roadmap for LoRAX. We'll continue to update this over time.

v0.10

• Speculative decoding adapters
• AQLM

v0.11

• Prefix caching
• BERT support
• Embedding endpoint
• Embedding adapters
• Classification adapters

Previous Releases

v0.9

• Adapter memory pool

Backlog

Models

• Llama
• Mistral
• GPT2
• Qwen
• Mixtral
• Phi
• Bloom
• BERT
• Stable-Diffusion

Adapters

• LoRA
• Classification Head
• Embedding MLP
• MoLoRA
• Medusa
• TART
• LCM-LoRA
• LoRA blending (multiple LoRAs per request)
• LongLoRA

Throughput / Latency

• Paged Attention v2
• Lookahead Decoding
• SGMV with variable ranks
• SGMV with tensor parallelism

Quantization

• bitsandbytes
• GPT-Q
• AWQ

Usability

• Prebuilt server wheels
• SkyPilot usage guide
• Example notebooks

[RileyCodes]

is AWQ supported?

[tgaddair]

Hey @RileyCodes, not yet, will add that to the roadmap!

[abhibst]

does we have tested bitsandbytes Quantization ?

[tgaddair]

Hey @abhibst, I've done some basic sanity checks on it, but haven't tested it very thoroughly. Please feel free to report any issues you encounter and I'll take a look!

[abhibst]

Sure Thanks for confirming

[sansavision]

How would you go about adding this in Stable Diffusion? I am really interested in experimenting with that.

[tgaddair]

Hey @sansavision, at a high level it would look a lot like the LoRA pipeline used in Diffusers: https://github.com/huggingface/api-inference-community/blob/main/docker_images/diffusers/app/pipelines/text_to_image.py#L25

A v0 shouldn't be too bad, we would basically just run a single forward pass to generate the image and perform postprocessing (as part of the existing Prefill step) and short-circuit the Decode step.

[flozi00]

If no one has started I will start working on awq tomorrow

[tgaddair]

Nice! Thanks @flozi00, that would be awesome!

[SamGalanakis]

Any plans to support vision transformers from huggingface / timm? A lot of potential use cases there for deploying many classifiers. If not what would that entail? Would be open to contributing if possible.

[tgaddair]

Hey @SamGalanakis, great suggestion! The plan at the moment is to start by supporting text classifiers. Once that framework is in place for that, it should be hopefully relatively straightforward to support image classifiers as well. Happy to start a thread on Discord to discuss!