Gkd trainer #1814
Gkd trainer #1814
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
There was a problem hiding this comment.
Thanks for ironing out the final kinks! I've tested Qwen and Gemma models and it seems to work well - we can always iterate a bit on the small details in follow up PRs.
I've left some final suggestions and have a question about the need to pass sft_args to the training vs just having a single set of args. Is it because the parent class won't accept the new params of the GKD config? If that's the case, then let's keep it as is and merge once you've incorporated the final changes
| self, | ||
| teacher_model: Union[PreTrainedModel, nn.Module, str], | ||
| args: Optional[GKDConfig] = None, | ||
| *sft_args, |
There was a problem hiding this comment.
Silly question, but why can't we just have args in the init since the GKDConfig already subclasses SFTConfig?
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
Co-authored-by: lewtun <lewis.c.tunstall@gmail.com>
|
Any idea why the doc build is failing @kashif ? |
|
not sure... i was also checking |
https://arxiv.org/pdf/2306.13649