Any plan to support cpu only mode?

[happy15]

Really impressive results 👏

Any plan to support cpu only mode? Thus it could be used on commodity laptops such as Macbook Pro.

[okpatil4u]

Or M1/M2 mac with metal support, ggml-org/llama.cpp#1642

[NewtonTrendy]

Is there a reason why not? Is it not possible to speed up CPU inference in the same ways?

[willtejeda]

+1 for metal support

[xiaotiancd]

This should dramatically reduce the cost if CPU acceleration is enabled, why not?
However, a CPU server usually has more memory than GPU, so the paging may not be a accelerating point, but vllm may have other approach to boost the performance, hope to see that soon.

[isandburn]

I have a CPU use case that may be of interest to many: having used Spark in the past, i have access to VM environments that have a ton of CPU and RAM (unfortunately, no GPUs and i can't control that)... i would love to reallocate a bunch of those resources to see how vLLM would be able to run on these preexisting (already bought and paid for) resources.

[pathorn]

I just made a PR #2244 - feel free to test and give feedback

I only tested on Apple Silicon (M3) but the original code from #1028 was designed for Intel, so recent Intel CPUs with bfloat16 instructions should also work. Let me know if I broke anything.

[sd3ntato]

hello, since I don't own any NVIDIA GPUs and can only deploy on paid, hosted solutions, I'd need cpu support for local development in order not to pay for development processes.
I don't think I am the only one.

[jyono]

also wondering...

[iamjameswalters]

Very interested in this!

[liwenju0]

Very interested too!

[TwoTwenty]

Please