Skip to content

Commit 1d646ba

Browse files
thomwolfthomwolf
committed
2 parents 9676d1a + 8349d75 commit 1d646ba

File tree

14 files changed

+361
-19
lines changed

14 files changed

+361
-19
lines changed

‎docs/source/index.rst

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,38 @@
11
Transformers
22
================================================================================================================================================
33

4-
Transformers is a library of state-of-the-art pre-trained models for Natural Language Processing (NLP).
4+
🤗 Transformers (formerly known as `pytorch-transformers` and `pytorch-pretrained-bert`) provides general-purpose architectures
5+
(BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet...) for Natural Language Understanding (NLU) and Natural Language Generation
6+
(NLG) with over 32+ pretrained models in 100+ languages and deep interoperability between TensorFlow 2.0 and PyTorch.
57

6-
The library currently contains PyTorch implementations, pre-trained model weights, usage scripts and conversion utilities for the following models:
8+
Features
9+
---------------------------------------------------
10+
11+
- As easy to use as pytorch-transformers
12+
- As powerful and concise as Keras
13+
- High performance on NLU and NLG tasks
14+
- Low barrier to entry for educators and practitioners
15+
16+
State-of-the-art NLP for everyone
17+
- Deep learning researchers
18+
- Hands-on practitioners
19+
- AI/ML/NLP teachers and educators
20+
21+
Lower compute costs, smaller carbon footprint
22+
- Researchers can share trained models instead of always retraining
23+
- Practitioners can reduce compute time and production costs
24+
- 8 architectures with over 30 pretrained models, some in more than 100 languages
25+
26+
Choose the right framework for every part of a model's lifetime
27+
- Train state-of-the-art models in 3 lines of code
28+
- Deep interoperability between TensorFlow 2.0 and PyTorch models
29+
- Move a single model between TF2.0/PyTorch frameworks at will
30+
- Seamlessly pick the right framework for training, evaluation, production
31+
32+
Contents
33+
---------------------------------
34+
35+
The library currently contains PyTorch and Tensorflow implementations, pre-trained model weights, usage scripts and conversion utilities for the following models:
736

837
1. `BERT <https://github.com/google-research/bert>`_ (from Google) released with the paper `BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding <https://arxiv.org/abs/1810.04805>`_ by Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova.
938
2. `GPT <https://github.com/openai/finetune-transformer-lm>`_ (from OpenAI) released with the paper `Improving Language Understanding by Generative Pre-Training <https://blog.openai.com/language-unsupervised>`_ by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever.
@@ -14,6 +43,7 @@ The library currently contains PyTorch implementations, pre-trained model weight
1443
7. `RoBERTa <https://github.com/pytorch/fairseq/tree/master/examples/roberta>`_ (from Facebook), released together with the paper a `Robustly Optimized BERT Pretraining Approach <https://arxiv.org/abs/1907.11692>`_ by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov.
1544
8. `DistilBERT <https://huggingface.co/transformers/model_doc/distilbert.html>`_ (from HuggingFace) released together with the blog post `Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT <https://medium.com/huggingface/distilbert-8cf3380435b5>`_ by Victor Sanh, Lysandre Debut and Thomas Wolf.
1645

46+
1747
.. toctree::
1848
:maxdepth: 2
1949
:caption: Notes
@@ -37,6 +67,7 @@ The library currently contains PyTorch implementations, pre-trained model weight
3767
main_classes/model
3868
main_classes/tokenizer
3969
main_classes/optimizer_schedules
70+
main_classes/processors
4071

4172
.. toctree::
4273
:maxdepth: 2

‎docs/source/main_classes/model.rst

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,3 +13,9 @@ The base class ``PreTrainedModel`` implements the common methods for loading/sav
1313

1414
.. autoclass:: transformers.PreTrainedModel
1515
:members:
16+
17+
``TFPreTrainedModel``
18+
~~~~~~~~~~~~~~~~~~~~~
19+
20+
.. autoclass:: pytorch_transformers.TFPreTrainedModel
21+
:members:

‎docs/source/main_classes/processors.rst

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
Processors
2+
----------------------------------------------------
3+
4+
This library includes processors for several traditional tasks. These processors can be used to process a dataset into
5+
examples that can be fed to a model.
6+
7+
Processors
8+
~~~~~~~~~~~~~~~~~~~~~
9+
10+
All processors follow the same architecture which is that of the
11+
:class:`~pytorch_transformers.data.processors.utils.DataProcessor`. The processor returns a list
12+
of :class:`~pytorch_transformers.data.processors.utils.InputExample`. These
13+
:class:`~pytorch_transformers.data.processors.utils.InputExample` can be converted to
14+
:class:`~pytorch_transformers.data.processors.utils.InputFeatures` in order to be fed to the model.
15+
16+
.. autoclass:: pytorch_transformers.data.processors.utils.DataProcessor
17+
:members:
18+
19+
20+
.. autoclass:: pytorch_transformers.data.processors.utils.InputExample
21+
:members:
22+
23+
24+
.. autoclass:: pytorch_transformers.data.processors.utils.InputFeatures
25+
:members:
26+
27+
28+
GLUE
29+
~~~~~~~~~~~~~~~~~~~~~
30+
31+
`General Language Understanding Evaluation (GLUE) <https://gluebenchmark.com/>`__ is a benchmark that evaluates
32+
the performance of models across a diverse set of existing NLU tasks. It was released together with the paper
33+
`GLUE: A multi-task benchmark and analysis platform for natural language understanding <https://openreview.net/pdf?id=rJ4km2R5t7>`__
34+
35+
This library hosts a total of 10 processors for the following tasks: MRPC, MNLI, MNLI (mismatched),
36+
CoLA, SST2, STSB, QQP, QNLI, RTE and WNLI.
37+
38+
Those processors are:
39+
- :class:`~pytorch_transformers.data.processors.utils.MrpcProcessor`
40+
- :class:`~pytorch_transformers.data.processors.utils.MnliProcessor`
41+
- :class:`~pytorch_transformers.data.processors.utils.MnliMismatchedProcessor`
42+
- :class:`~pytorch_transformers.data.processors.utils.Sst2Processor`
43+
- :class:`~pytorch_transformers.data.processors.utils.StsbProcessor`
44+
- :class:`~pytorch_transformers.data.processors.utils.QqpProcessor`
45+
- :class:`~pytorch_transformers.data.processors.utils.QnliProcessor`
46+
- :class:`~pytorch_transformers.data.processors.utils.RteProcessor`
47+
- :class:`~pytorch_transformers.data.processors.utils.WnliProcessor`
48+
49+
Additionally, the following method can be used to load values from a data file and convert them to a list of
50+
:class:`~pytorch_transformers.data.processors.utils.InputExample`.
51+
52+
.. automethod:: pytorch_transformers.data.processors.glue.glue_convert_examples_to_features
53+
54+
Example usage
55+
^^^^^^^^^^^^^^^^^^^^^^^^^
56+
57+
An example using these processors is given in the
58+
`run_glue.py <https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py>`__ script.

‎docs/source/model_doc/bert.rst

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,3 +70,59 @@ BERT
7070
.. autoclass:: transformers.BertForQuestionAnswering
7171
:members:
7272

73+
74+
``TFBertModel``
75+
~~~~~~~~~~~~~~~~~~~~
76+
77+
.. autoclass:: pytorch_transformers.TFBertModel
78+
:members:
79+
80+
81+
``TFBertForPreTraining``
82+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
83+
84+
.. autoclass:: pytorch_transformers.TFBertForPreTraining
85+
:members:
86+
87+
88+
``TFBertForMaskedLM``
89+
~~~~~~~~~~~~~~~~~~~~~~~~~~
90+
91+
.. autoclass:: pytorch_transformers.TFBertForMaskedLM
92+
:members:
93+
94+
95+
``TFBertForNextSentencePrediction``
96+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
97+
98+
.. autoclass:: pytorch_transformers.TFBertForNextSentencePrediction
99+
:members:
100+
101+
102+
``TFBertForSequenceClassification``
103+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
104+
105+
.. autoclass:: pytorch_transformers.TFBertForSequenceClassification
106+
:members:
107+
108+
109+
``TFBertForMultipleChoice``
110+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
111+
112+
.. autoclass:: pytorch_transformers.TFBertForMultipleChoice
113+
:members:
114+
115+
116+
``TFBertForTokenClassification``
117+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
118+
119+
.. autoclass:: pytorch_transformers.TFBertForTokenClassification
120+
:members:
121+
122+
123+
``TFBertForQuestionAnswering``
124+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
125+
126+
.. autoclass:: pytorch_transformers.TFBertForQuestionAnswering
127+
:members:
128+

‎docs/source/model_doc/distilbert.rst

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,30 @@ DistilBERT
4141

4242
.. autoclass:: transformers.DistilBertForQuestionAnswering
4343
:members:
44+
45+
``TFDistilBertModel``
46+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
47+
48+
.. autoclass:: pytorch_transformers.TFDistilBertModel
49+
:members:
50+
51+
52+
``TFDistilBertForMaskedLM``
53+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
54+
55+
.. autoclass:: pytorch_transformers.TFDistilBertForMaskedLM
56+
:members:
57+
58+
59+
``TFDistilBertForSequenceClassification``
60+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
61+
62+
.. autoclass:: pytorch_transformers.TFDistilBertForSequenceClassification
63+
:members:
64+
65+
66+
``TFDistilBertForQuestionAnswering``
67+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
68+
69+
.. autoclass:: pytorch_transformers.TFDistilBertForQuestionAnswering
70+
:members:

‎docs/source/model_doc/gpt.rst

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,3 +34,24 @@ OpenAI GPT
3434

3535
.. autoclass:: transformers.OpenAIGPTDoubleHeadsModel
3636
:members:
37+
38+
39+
``TFOpenAIGPTModel``
40+
~~~~~~~~~~~~~~~~~~~~~~~~~
41+
42+
.. autoclass:: pytorch_transformers.TFOpenAIGPTModel
43+
:members:
44+
45+
46+
``TFOpenAIGPTLMHeadModel``
47+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
48+
49+
.. autoclass:: pytorch_transformers.TFOpenAIGPTLMHeadModel
50+
:members:
51+
52+
53+
``TFOpenAIGPTDoubleHeadsModel``
54+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
55+
56+
.. autoclass:: pytorch_transformers.TFOpenAIGPTDoubleHeadsModel
57+
:members:

‎docs/source/model_doc/gpt2.rst

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,3 +34,24 @@ OpenAI GPT2
3434

3535
.. autoclass:: transformers.GPT2DoubleHeadsModel
3636
:members:
37+
38+
39+
``TFGPT2Model``
40+
~~~~~~~~~~~~~~~~~~~~~
41+
42+
.. autoclass:: pytorch_transformers.TFGPT2Model
43+
:members:
44+
45+
46+
``TFGPT2LMHeadModel``
47+
~~~~~~~~~~~~~~~~~~~~~~~~~~~
48+
49+
.. autoclass:: pytorch_transformers.TFGPT2LMHeadModel
50+
:members:
51+
52+
53+
``TFGPT2DoubleHeadsModel``
54+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
55+
56+
.. autoclass:: pytorch_transformers.TFGPT2DoubleHeadsModel
57+
:members:

‎docs/source/model_doc/roberta.rst

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,3 +34,24 @@ RoBERTa
3434

3535
.. autoclass:: transformers.RobertaForSequenceClassification
3636
:members:
37+
38+
39+
``TFRobertaModel``
40+
~~~~~~~~~~~~~~~~~~~~
41+
42+
.. autoclass:: pytorch_transformers.TFRobertaModel
43+
:members:
44+
45+
46+
``TFRobertaForMaskedLM``
47+
~~~~~~~~~~~~~~~~~~~~~~~~~~
48+
49+
.. autoclass:: pytorch_transformers.TFRobertaForMaskedLM
50+
:members:
51+
52+
53+
``TFRobertaForSequenceClassification``
54+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
55+
56+
.. autoclass:: pytorch_transformers.TFRobertaForSequenceClassification
57+
:members:

‎docs/source/model_doc/transformerxl.rst

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,3 +28,17 @@ Transformer XL
2828

2929
.. autoclass:: transformers.TransfoXLLMHeadModel
3030
:members:
31+
32+
33+
``TFTransfoXLModel``
34+
~~~~~~~~~~~~~~~~~~~~~~~~~~
35+
36+
.. autoclass:: pytorch_transformers.TFTransfoXLModel
37+
:members:
38+
39+
40+
``TFTransfoXLLMHeadModel``
41+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
42+
43+
.. autoclass:: pytorch_transformers.TFTransfoXLLMHeadModel
44+
:members:

‎docs/source/model_doc/xlm.rst

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,3 +39,31 @@ XLM
3939

4040
.. autoclass:: transformers.XLMForQuestionAnswering
4141
:members:
42+
43+
44+
``TFXLMModel``
45+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
46+
47+
.. autoclass:: pytorch_transformers.TFXLMModel
48+
:members:
49+
50+
51+
``TFXLMWithLMHeadModel``
52+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
53+
54+
.. autoclass:: pytorch_transformers.TFXLMWithLMHeadModel
55+
:members:
56+
57+
58+
``TFXLMForSequenceClassification``
59+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
60+
61+
.. autoclass:: pytorch_transformers.TFXLMForSequenceClassification
62+
:members:
63+
64+
65+
``TFXLMForQuestionAnsweringSimple``
66+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
67+
68+
.. autoclass:: pytorch_transformers.TFXLMForQuestionAnsweringSimple
69+
:members:

‎docs/source/model_doc/xlnet.rst

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,31 @@ XLNet
4141

4242
.. autoclass:: transformers.XLNetForQuestionAnswering
4343
:members:
44+
45+
46+
``TFXLNetModel``
47+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
48+
49+
.. autoclass:: pytorch_transformers.TFXLNetModel
50+
:members:
51+
52+
53+
``TFXLNetLMHeadModel``
54+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
55+
56+
.. autoclass:: pytorch_transformers.TFXLNetLMHeadModel
57+
:members:
58+
59+
60+
``TFXLNetForSequenceClassification``
61+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
62+
63+
.. autoclass:: pytorch_transformers.TFXLNetForSequenceClassification
64+
:members:
65+
66+
67+
``TFXLNetForQuestionAnsweringSimple``
68+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
69+
70+
.. autoclass:: pytorch_transformers.TFXLNetForQuestionAnsweringSimple
71+
:members:

‎docs/source/pretrained_models.rst

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,15 +44,15 @@ Here is the full list of the currently provided pretrained models together with
4444
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
4545
| | ``bert-large-uncased-whole-word-masking-finetuned-squad`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. |
4646
| | | | The ``bert-large-uncased-whole-word-masking`` model fine-tuned on SQuAD |
47-
| | | (see details of fine-tuning in the `example section <https://github.com/huggingface/transformers/tree/master/examples>`__). |
47+
| | | (see details of fine-tuning in the `example section <https://github.com/huggingface/transformers/tree/master/examples>`__). |
4848
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
4949
| | ``bert-large-cased-whole-word-masking-finetuned-squad`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters |
5050
| | | | The ``bert-large-cased-whole-word-masking`` model fine-tuned on SQuAD |
51-
| | | (see `details of fine-tuning in the example section <https://huggingface.co/transformers/examples.html>`__) |
51+
| | | (see `details of fine-tuning in the example section <https://huggingface.co/transformers/examples.html>`__) |
5252
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
5353
| | ``bert-base-cased-finetuned-mrpc`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
5454
| | | | The ``bert-base-cased`` model fine-tuned on MRPC |
55-
| | | (see `details of fine-tuning in the example section <https://huggingface.co/transformers/examples.html>`__) |
55+
| | | (see `details of fine-tuning in the example section <https://huggingface.co/transformers/examples.html>`__) |
5656
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
5757
| GPT | ``openai-gpt`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
5858
| | | | OpenAI GPT English model |

0 commit comments

Comments
 (0)