-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add T5 model #916
Add T5 model #916
Conversation
@yingyibiao @JunnYu 这个PR还有效吗? |
有效 |
需要在example/language_model下面新增一个案例 |
权重已上传~ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PreTrained -> Pretrained
统一一下这个单词的命名。
super().__init__() | ||
self.t5 = t5 | ||
del self.t5.decoder | ||
paddle.device.cuda.empty_cache() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
T5EncoderModel还是参考hf的实现来写;目前这种写法会存在一个显存占用的脉冲,容易造成oom。
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
现在换成了这种写法,可以吗?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这个类建议按照参考T5Model的方式来初始化(传入构造T5EncoderModel的具体参数)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
'T5Model', | ||
"T5PretrainedModel", | ||
'T5ForConditionalGeneration', | ||
'T5EncoderModel', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这里也删除一下
飞桨论文复现挑战赛(第四期)Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer 论文复现提交。