Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix disabling GPU use #56

Merged
merged 2 commits into from
Aug 24, 2024
Merged

Fix disabling GPU use #56

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cfbf0d4
Fix typo: disableGPU != disableGpu
jehna Aug 24, 2024
7fb58d4
Add disabling GPU altogether
jehna Aug 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/commands/local.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,11 @@ export const local = cli()
verbose.enabled = true;
}

verbose.log("Starting local inference with options: ", opts);

const prompt = await llama({
model: opts.model,
disableGPU: opts.disableGPU,
disableGpu: opts.disableGpu,
Comment on lines -30 to +32
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the TypeScript typing have pointed out this issue earlier?

Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but unfortunately the commander package just passes any as the options. yargs was better in making the options strongly typed, although I found the commander package a bit more ergonomic to use. I'm thinking about changing away from commander though, exactly because this should have been a compiler-caught bug.

Copy link

@0xdevalias 0xdevalias Aug 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah true; yeah, that makes sense.

Does commander let you override the options type? It still could have let the bug through if you made the bug in those types as well, but at least it would stop the 'any' propagating further.

seed: opts.seed ? parseInt(opts.seed) : undefined
});
await unminify(filename, opts.outputDir, [
Expand Down
21 changes: 15 additions & 6 deletions src/plugins/local-llm-rename/llama.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
import { getLlama, LlamaChatSession, LlamaGrammar } from "node-llama-cpp";
import {
getLlama,
LlamaChatSession,
LlamaGrammar,
LlamaModelOptions
} from "node-llama-cpp";
import { Gbnf } from "./gbnf.js";
import { getModelPath, getModelWrapper } from "../../local-models.js";
import { verbose } from "../../verbose.js";

export type Prompt = (
systemPrompt: string,
Expand All @@ -13,13 +19,16 @@ const IS_CI = process.env["CI"] === "true";
export async function llama(opts: {
seed?: number;
model: string;
disableGPU?: boolean;
disableGpu?: boolean;
}): Promise<Prompt> {
const llama = await getLlama();
const model = await llama.loadModel({
const disableGpu = opts.disableGpu ?? IS_CI;
const llama = await getLlama({ gpu: disableGpu ? false : "auto" });
const modelOpts: LlamaModelOptions = {
modelPath: getModelPath(opts?.model),
gpuLayers: (opts?.disableGPU ?? IS_CI) ? 0 : undefined
});
gpuLayers: disableGpu ? 0 : undefined
Copy link

@0xdevalias 0xdevalias Aug 26, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, if GPU is disabled in the call to getLlama now, do you actually need to set gpuLayers to 0 here? Or is this just leftover from your previous attempted approach to disable GPU?

};
verbose.log("Loading model with options", modelOpts);
const model = await llama.loadModel(modelOpts);

const context = await model.createContext({ seed: opts?.seed });

Expand Down