Skip to content

Commit

Permalink
Add peft generation example (#1427)
Browse files Browse the repository at this point in the history
  • Loading branch information
sywangyi authored Oct 17, 2024
1 parent 0b981a9 commit e7f97ad
Showing 1 changed file with 16 additions and 0 deletions.
16 changes: 16 additions & 0 deletions examples/text-generation/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,22 @@ python run_generation.py \

> The prompt length is limited to 16 tokens. Prompts longer than this will be truncated.
### Use PEFT models for generation

You can also provide the path to a PEFT model to perform generation with the argument `--peft_model`.

For example:
```bash
python run_generation.py \
--model_name_or_path meta-llama/Llama-2-7b-hf \
--use_hpu_graphs \
--use_kv_cache \
--batch_size 1 \
--bf16 \
--max_new_tokens 100 \
--prompt "Here is my prompt" \
--peft_model yard1/llama-2-7b-sql-lora-test
```

### Using growing bucket optimization

Expand Down

0 comments on commit e7f97ad

Please sign in to comment.