Add FNet #1499
Add FNet #1499
Conversation
|
|
预训练模型位置1.spiece.model
2.model_state.pdparams |
|
@HJHGJGHHG 感谢贡献，麻烦签署一下CLA哈 #1499 (comment) |
|
多谢提醒，已签署。
…------------------ 原始邮件 ------------------
发件人: "PaddlePaddle/PaddleNLP" ***@***.***>;
发送时间: 2021年12月23日(星期四) 晚上10:06
***@***.***>;
***@***.******@***.***>;
主题: Re: [PaddlePaddle/PaddleNLP] Add FNet (PR #1499)
@HJHGJGHHG 感谢贡献，麻烦签署一下CLA哈 #1499 (comment)
—
Reply to this email directly, view it on GitHub, or unsubscribe.
Triage notifications on the go with GitHub Mobile for iOS or Android.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
| @@ -0,0 +1,871 @@ | |||
| # encoding=utf8 | |||
There was a problem hiding this comment.
paddlenlp 仅支持python3.6+，不需要指定编码。可以删去
| return x * F.sigmoid(x) | ||
|
|
||
|
|
||
| def gelu_new(x): |
There was a problem hiding this comment.
这个GELU实现和paddle.nn.funtional.gelu()实现有什么不同？
paddle.nn.funtional.gelu()是否可以满足需求？
There was a problem hiding this comment.
是paddle.nn.funtional.gelu()在approximate=True时的版本。由于不好指定关键字参数所以此处单独列出，其他模型（如albert）也如此。后续可将paddle.nn.funtional.gelu()拆成两部分，更为精简。
| intermediate_output = self.intermediate(fourier_output) | ||
| layer_output = self.output(intermediate_output, fourier_output) | ||
|
|
||
| return (layer_output,) |
| return {"last_hidden_state": hidden_states, | ||
| "all_hidden_states": all_hidden_states | ||
| } | ||
| return (hidden_states,) |
|
|
||
| def forward(self, hidden_states): | ||
| outputs = self.fourier_transform(hidden_states).real() | ||
| return (outputs,) |
| def forward(self, hidden_states): | ||
| self_outputs = self.fourier_transform(hidden_states) | ||
| fourier_output = self.output(self_outputs[0], hidden_states) | ||
| return (fourier_output,) |
| @@ -0,0 +1,291 @@ | |||
| # encoding=utf8 | |||
| def __init__(self): | ||
| super().__init__() | ||
| # self.fourier_transform = paddle.fft.fftn | ||
| self.fourier_transform = partial(paddle.fft.fftn, axes=(1, 2)) |
There was a problem hiding this comment.
paddle.fft.fftn()不建议用partial封装，使用partial封装会可能导致出错堆栈信息复杂。直接调用即可。
| @@ -0,0 +1,871 @@ | |||
| # encoding=utf8 | |||
| return x * F.sigmoid(x) | ||
|
|
||
|
|
||
| def gelu_new(x): |
There was a problem hiding this comment.
是paddle.nn.funtional.gelu()在approximate=True时的版本。由于不好指定关键字参数所以此处单独列出，其他模型（如albert）也如此。后续可将paddle.nn.funtional.gelu()拆成两部分，更为精简。
|
参考其他模型给modeling.py和tokenizer.py这两个文件添加docstrings |
可以提供百度网盘或者google_drive下载链接吗～ |
我重新上传一下~~ |
|
已上传至百度网盘 |
PR types
PR changes
Description
添加FNet模型文件