Supply Local Weights to an LLM instead of Downloading Weights from HuggingFace

[sgupta1007]

🚀 The feature, motivation and pitch

I am having local copy of llama weights and i want to supply those weights to create a chat application.Please include a CLI flag to do so

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

[mikekgfb]

This already exists, but is not documented in the master README.md which is targeted at more novice users.

There are some examples how to use local checkpoints in

torchchat/docs/ADVANCED-USERS.md

Line 245 in 019f76f

python3 torchchat.py generate [--compile] --checkpoint-path ${MODEL_PATH} --prompt "Hello, my name is" --device [ cuda | mps | cpu ]

and #1336 has some examples used during granite support development. That issue is a great guide to end-to-end adding new features and support for a model that needs extra options beyond the base llama-inspired model types.

You'll need
(1) a checkpoint,
(2) a tokenizer json,
(3) (a) a parameter file, or (b) specify one of the existing parameter files with --params-table or (c) partial-match by pathname with the same algorithm as GPT-fast

The model can pick up a tokenizer file automatically if it's stored in the same directory, and the parameter file via (3c) pathname matching to recognize a few well-known model variants. (Like it does in the stories15M example in the advanced users guide.)

Please see #1336 as example:

# Run with direct reference to artifacts
python torchchat.py generate \
  --prompt "Write a python function to sort numbers and strings with numeric prefixes" \
  --checkpoint-path $HOME/models/ibm-granite/granite-3b-code-instruct-128k/model.pth \
  --tokenizer-path $HOME/models/ibm-granite/granite-3b-code-instruct-128k/tokenizer.json \
  --params-path torchchat/model_params/Granite-3B-Code.json

Check out

torchchat/torchchat/cli/cli.py

Line 466 in 019f76f

"--params-path",

for a list of other options that may be of interest for you.

See also description in https://docs.google.com/document/d/1WvbVM1liEBaePltydkYkjFXnCwhe0yeuK-OtTlur9FA/edit?usp=sharing

[Jack-Khuu]

Thanks for answering @mikekgfb and spinning up a doc!! It'll be great for future users

I'm also planning to send out an RFC after the holidays for an improve flow so stay tuned :)