Fix disabling GPU use #56
Conversation
This made the local inference try to use the GPU even when the user provided the `--disableGpu` flag.
Setting the layers to 0 seemed to do most of the work, but it still used some of the available GPU. Which was weird. Let's see if this messes up the CI test times.
|
No effect in CI test runs, this is good to go 🚀 |
| disableGPU: opts.disableGPU, | ||
| disableGpu: opts.disableGpu, |
There was a problem hiding this comment.
Should the TypeScript typing have pointed out this issue earlier?
There was a problem hiding this comment.
Yes, but unfortunately the commander package just passes any as the options. yargs was better in making the options strongly typed, although I found the commander package a bit more ergonomic to use. I'm thinking about changing away from commander though, exactly because this should have been a compiler-caught bug.
There was a problem hiding this comment.
Ah true; yeah, that makes sense.
Does commander let you override the options type? It still could have let the bug through if you made the bug in those types as well, but at least it would stop the 'any' propagating further.
| modelPath: getModelPath(opts?.model), | ||
| gpuLayers: (opts?.disableGPU ?? IS_CI) ? 0 : undefined | ||
| }); | ||
| gpuLayers: disableGpu ? 0 : undefined |
There was a problem hiding this comment.
Curious, if GPU is disabled in the call to getLlama now, do you actually need to set gpuLayers to 0 here? Or is this just leftover from your previous attempted approach to disable GPU?
|
Right now without GPU it just do not work at all |
|
That's weird. I do run CI tests without GPU and they seem to be working fine. Do you get any errors? Running even a small model on a mid renage CPU is super slow though. |
|
Maybe because I am using an old laptop, I don't have access to another for the moment... No, I didn't get any error... I would like to know what is the problem in order to try to fix it. Yes, I am aware of LLM being extremely slow as I try to use AnythingLLM to try some uncensored models... |
|
Does the program exit by itself or does it just seem to hang forever? If it exits, what's the exit code? |
Yes it exists just after I launch it... Do you want a video? |
|
I managed to make this work via WSL by installing all the required components (I have to write a tutorial about this, but I need to setup another machine to start from scratch and take note of all steps)... So for the moment you can just forget about this issue... Side note : I will add support in the code "disableGPU" as well as I think it is more appropriate to write it and do PR. |
Fixes #55
Need to check what happens to test run times if e161521 is applied, best case they stay at ~5min