You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#1464
This posts discuss fine-tuning GPT2,
With GPT2/LLaMA, by default, we need to input the [prompt label] the whole sentence model([prompt label]) in fine-tuning and caculate the CrossEntropy on the label part, and the model output the model().logits.
Are there any ways to input the prompt only and do the fine-tuning in the seq2seq manner? (model(prompt)), this way we minimize the loss of log p(y|x).
Get the feature of model(prompt) rather than model([prompt label]) is the whole point.
Motivation
seq2seq equivalence fine-tuning workflow for decoder-only transformers.
Your contribution
I could submit PR with discussion as a guidance.
The text was updated successfully, but these errors were encountered:
Hey 🤗 thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the forum instead? I'm sure the community will be of help!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Feature request
#1464
This posts discuss fine-tuning GPT2,
With GPT2/LLaMA, by default, we need to input the [prompt label] the whole sentence
model([prompt label])
in fine-tuning and caculate the CrossEntropy on the label part, and the model output the model().logits.Are there any ways to input the prompt only and do the fine-tuning in the seq2seq manner? (
model(prompt)
), this way we minimize the loss of log p(y|x).Get the feature of
model(prompt)
rather thanmodel([prompt label])
is the whole point.Motivation
seq2seq equivalence fine-tuning workflow for decoder-only transformers.
Your contribution
I could submit PR with discussion as a guidance.
The text was updated successfully, but these errors were encountered: