-
Notifications
You must be signed in to change notification settings - Fork 7.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Support more than 64 CPU threads #2696
Comments
I'm not from Nomic, but I have to ask what is the benefit of even 64 CPU threads? Have you benchmarked 64 threads vs 32, 16, or even 8, and found that higher (after a certain point) are better? My understanding, and shown in my own tests, is that after 6-8 cpu threads, the memory bus is saturated, and more threads tend to do nothing. Maybe you can do a few more than that on Epyc, just don't otherwise expect much more than that to actually accomplish anything. If I am wrong, would appreciate learning from tests you have done. |
Looks like code is here: gpt4all/gpt4all-chat/mysettings.cpp Lines 434 to 443 in 56d5a23
Which means, the thread count is determined by what Qt thinks should be the upper limit. I'm unsure whether you'd get more performance out of it with a higher value, in any case. |
Maybe, although I don't know about Qt internals and that Q&A is really old. But anyway, as chrisbarrera said, I'm not even sure it would help any to go past Qt defined limit. |
llama.cpp on CPU is memory-bottlenecked in practice, so using more CPU threads doesn't provide much benefit. The default of 4 threads is enough on my machine. Try with ollama or the llama.cpp CLI and see if you actually get any t/s improvement compared to 64 threads—you may actually see a slowdown. |
Feature Request
currently gpt4all settings allow setting max cpu threads to 64 only
when i set it to 192 (which is my current hw setup) it always reverts to 64
ollama by itself supports 192 - i tried that so maybe this is just some UI restriction? didn't look in the code
thanks
The text was updated successfully, but these errors were encountered: