funnel transformer #1419
funnel transformer #1419
Conversation
| different token than `bos`, the id of that token. | ||
| - **sep_token_id** (:obj:`int`, `optional`)) -- The id of the `separation` token. | ||
|
|
||
| PyTorch specific parameters |
There was a problem hiding this comment.
这里是不需要提供PyTorch和TF的特定的参数，可以去掉
| return x | ||
|
|
||
| def constant_(x, val): | ||
| temp_value=paddle.zeros_like(x)+val |
There was a problem hiding this comment.
是的， 已经改为 paddle.full_like(x,fill_value=val)
| return x * paddle.sigmoid(x) | ||
|
|
||
|
|
||
| if version.parse(paddle.__version__) < version.parse("1.7"): |
There was a problem hiding this comment.
没有作用， 之前torch的版本检验，已经remove。 保留 silu = nn.functional.silu
|
|
||
|
|
||
|
|
||
| def expand(self, *sizes): |
| return x | ||
|
|
||
|
|
||
| def repeat_interleave(x, repeats, dim=None): |
| return paddle.reshape(x, orig_shape) | ||
|
|
||
|
|
||
| def gather(x, dim, index): |
|
|
||
| def is_tensor(x): | ||
| """ | ||
| Tests if ``x`` is a :obj:`paddle.Tensor`, :obj:`tf.Tensor`, obj:`jaxlib.xla_extension.DeviceArray` or |
| return_dict=None, | ||
| ): | ||
| r""" | ||
| labels (:obj:`paddle.LongTensor` of shape :obj:`(batch_size,)`, `optional`): |
There was a problem hiding this comment.
这里以及下面几个任务类都可以稍微整理一下注释，可以参考BertForSequenceClassification。还有就是paddle只有Tensor没有LongTensor类型，注意修改~
| return_dict=None, | ||
| ): | ||
| r""" | ||
| labels (``paddle.LongTensor`` of shape ``(batch_size, sequence_length)``, `optional`): |
| return self.linear_out(hidden) | ||
|
|
||
| from collections import OrderedDict | ||
| class FunnelForPreTrainingOutput(OrderedDict): |
There was a problem hiding this comment.
注释里涉及到FloatTenor,LongTensor类型的统一修改为Tensor
| @@ -86,3 +86,5 @@ | |||
| from .reformer.tokenizer import * | |||
| from .mobilebert.modeling import * | |||
| from .mobilebert.tokenizer import * | |||
| return ACT2FN[activation_string] | ||
| else: | ||
| raise KeyError(f"function {activation_string} not found in ACT2FN mapping {list(ACT2FN.keys())}") | ||
|
|
There was a problem hiding this comment.
这里空行较多，是否没有安装代码检查工具呢？建议安装git hook,并重新提交代码
pre-commit install
There was a problem hiding this comment.
不太会用，我尝试过window， 和ubuntu安装， commit的时候都出现先配置错误。 可否帮忙看看现在的版本是否还有空行问题？
|
|
||
| PyTorch specific parameters | ||
|
|
||
| - **torchscript** (:obj:`bool`, `optional`, defaults to :obj:`False`) -- Whether or not the model should be |
There was a problem hiding this comment.
貌似很多注释都出现torch、TensorFlow的字眼，目前PaddleNLP暂时不用支持torch和tensorflow的输入，可以把相关的信息删除掉
PR types
new transformer model
PR changes
主要是添加了funnel 模型在paddlenlp的transformer 下面， 和修改了run_squad.py 在examples 下，可以测试funnel 模型
Description
复现论文比赛第四期， funnel 的复现，团队：小木桌
请在google drive 下载所有pretrain 模型 放到 https://bj.bcebos.com/paddlenlp/models：
https://drive.google.com/drive/folders/1eo2Jq0xDd7qO_9-N5lf42WsqQSi7i9aS?usp=sharing