You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Update LED model card
* Remove extra arguments
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
The LED model was proposed in [Longformer: The Long-Document Transformer](https://huggingface.co/papers/2004.05150) by Iz
27
-
Beltagy, Matthew E. Peters, Arman Cohan.
28
-
29
-
The abstract from the paper is the following:
30
-
31
-
*Transformer-based models are unable to process long sequences due to their self-attention operation, which scales
32
-
quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention
33
-
mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or
34
-
longer. Longformer's attention mechanism is a drop-in replacement for the standard self-attention and combines a local
35
-
windowed attention with a task motivated global attention. Following prior work on long-sequence transformers, we
36
-
evaluate Longformer on character-level language modeling and achieve state-of-the-art results on text8 and enwik8. In
37
-
contrast to most prior work, we also pretrain Longformer and finetune it on a variety of downstream tasks. Our
38
-
pretrained Longformer consistently outperforms RoBERTa on long document tasks and sets new state-of-the-art results on
39
-
WikiHop and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a Longformer variant for supporting
40
-
long document generative sequence-to-sequence tasks, and demonstrate its effectiveness on the arXiv summarization
41
-
dataset.*
42
-
43
-
## Usage tips
44
-
45
-
-[`LEDForConditionalGeneration`] is an extension of
46
-
[`BartForConditionalGeneration`] exchanging the traditional *self-attention* layer with
47
-
*Longformer*'s *chunked self-attention* layer. [`LEDTokenizer`] is an alias of
48
-
[`BartTokenizer`].
49
-
- LED works very well on long-range *sequence-to-sequence* tasks where the `input_ids` largely exceed a length of
50
-
1024 tokens.
51
-
- LED pads the `input_ids` to be a multiple of `config.attention_window` if required. Therefore a small speed-up is
52
-
gained, when [`LEDTokenizer`] is used with the `pad_to_multiple_of` argument.
53
-
- LED makes use of *global attention* by means of the `global_attention_mask` (see
54
-
[`LongformerModel`]). For summarization, it is advised to put *global attention* only on the first
55
-
`<s>` token. For question answering, it is advised to put *global attention* on all tokens of the question.
56
-
- To fine-tune LED on all 16384, *gradient checkpointing* can be enabled in case training leads to out-of-memory (OOM)
57
-
errors. This can be done by executing `model.gradient_checkpointing_enable()`.
58
-
Moreover, the `use_cache=False`
59
-
flag can be used to disable the caching mechanism to save memory.
60
-
- LED is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
61
-
the left.
62
-
63
-
This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).
24
+
# LED
25
+
26
+
[Longformer-Encoder-Decoder (LED)](https://huggingface.co/papers/2004.05150) is an encoder-decoder transformer model for sequence-to-sequence tasks like summarization. It extends [Longformer](.longformer), an encoder-only model designed to handle long inputs, by adding a decoder layer. The decoder uses full self-attention on the encoded tokens and previously decoded locations. Because of Longformer's linear self-attention mechanism, LED is more efficient than standard encoder-decoder models when processing long sequences.
27
+
28
+
You can find all the original [LED] checkpoints under the [Ai2](https://huggingface.co/allenai/models?search=led) organization.
29
+
30
+
> [!TIP]
31
+
> This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).
32
+
>
33
+
> Click on the LED models in the right sidebar for more examples of how to apply LED to different language tasks.
34
+
35
+
The example below demonstrates how to summarize text with [`Pipeline`], [`AutoModel`], and from the command line.
36
+
37
+
<hfoptionsid="usage">
38
+
<hfoptionid="Pipeline">
39
+
40
+
```python
41
+
import torch
42
+
from transformers import pipeline
43
+
44
+
pipeline = pipeline(
45
+
task="summarization",
46
+
model="allenai/led-base-16384",
47
+
torch_dtype=torch.float16,
48
+
device=0
49
+
)
50
+
pipeline("""Plants are among the most remarkable and essential life forms on Earth, possessing a unique ability to produce their own food through a process known as photosynthesis. This complex biochemical process is fundamental not only to plant life but to virtually all life on the planet.
51
+
Through photosynthesis, plants capture energy from sunlight using a green pigment called chlorophyll, which is located in specialized cell structures called chloroplasts. In the presence of light, plants absorb carbon dioxide from the atmosphere through small pores in their leaves called stomata, and take in water from the soil through their root systems.
52
+
These ingredients are then transformed into glucose, a type of sugar that serves as a source of chemical energy, and oxygen, which is released as a byproduct into the atmosphere. The glucose produced during photosynthesis is not just used immediately; plants also store it as starch or convert it into other organic compounds like cellulose, which is essential for building their cellular structure.
53
+
This energy reserve allows them to grow, develop leaves, produce flowers, bear fruit, and carry out various physiological processes throughout their lifecycle.""")
54
+
```
55
+
56
+
</hfoption>
57
+
<hfoptionid="AutoModel">
58
+
59
+
```python
60
+
import torch
61
+
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
62
+
63
+
tokenizer = AutoTokenizer.from_pretrained(
64
+
"allenai/led-base-16384"
65
+
)
66
+
model = AutoModelForSeq2SeqLM.from_pretrained(
67
+
"allenai/led-base-16384",
68
+
torch_dtype=torch.float16,
69
+
device_map="auto"
70
+
)
71
+
72
+
input_text ="""Plants are among the most remarkable and essential life forms on Earth, possessing a unique ability to produce their own food through a process known as photosynthesis. This complex biochemical process is fundamental not only to plant life but to virtually all life on the planet.
73
+
Through photosynthesis, plants capture energy from sunlight using a green pigment called chlorophyll, which is located in specialized cell structures called chloroplasts. In the presence of light, plants absorb carbon dioxide from the atmosphere through small pores in their leaves called stomata, and take in water from the soil through their root systems.
74
+
These ingredients are then transformed into glucose, a type of sugar that serves as a source of chemical energy, and oxygen, which is released as a byproduct into the atmosphere. The glucose produced during photosynthesis is not just used immediately; plants also store it as starch or convert it into other organic compounds like cellulose, which is essential for building their cellular structure.
75
+
This energy reserve allows them to grow, develop leaves, produce flowers, bear fruit, and carry out various physiological processes throughout their lifecycle."""
!echo -e "Plants are among the most remarkable and essential life forms on Earth, possessing a unique ability to produce their own food through a process known as photosynthesis. This complex biochemical process is fundamental not only to plant life but to virtually all life on the planet. Through photosynthesis, plants capture energy from sunlight using a green pigment called chlorophyll, which is located in specialized cell structures called chloroplasts."| transformers-cli run --task summarization --model allenai/led-base-16384 --device 0
91
+
```
92
+
</hfoption>
93
+
</hfoptions>
94
+
95
+
Quantization reduces the memory burden of large models by representing the weights in a lower precision. Refer to the [Quantization](../quantization/overview) overview for more available quantization backends.
96
+
97
+
The example below uses [bitsandbytes](../quantization/bitsandbytes) to only quantize the weights to int4.
98
+
99
+
```python
100
+
import torch
101
+
from transformers import BitsAndBytesConfig, AutoModelForSeq2SeqLM, AutoTokenizer
102
+
103
+
quantization_config = BitsAndBytesConfig(
104
+
load_in_4bit=True,
105
+
bnb_4bit_compute_dtype=torch.bfloat16,
106
+
bnb_4bit_quant_type="nf4"
107
+
)
108
+
model = AutoModelForSeq2SeqLM.from_pretrained(
109
+
"allenai/led-large-16384",
110
+
torch_dtype=torch.bfloat16,
111
+
device_map="auto",
112
+
quantization_config=quantization_config
113
+
)
114
+
115
+
tokenizer = AutoTokenizer.from_pretrained(
116
+
"allenai/led-large-16384"
117
+
)
118
+
119
+
input_text ="""Plants are among the most remarkable and essential life forms on Earth, possessing a unique ability to produce their own food through a process known as photosynthesis. This complex biochemical process is fundamental not only to plant life but to virtually all life on the planet.
120
+
Through photosynthesis, plants capture energy from sunlight using a green pigment called chlorophyll, which is located in specialized cell structures called chloroplasts. In the presence of light, plants absorb carbon dioxide from the atmosphere through small pores in their leaves called stomata, and take in water from the soil through their root systems.
121
+
These ingredients are then transformed into glucose, a type of sugar that serves as a source of chemical energy, and oxygen, which is released as a byproduct into the atmosphere. The glucose produced during photosynthesis is not just used immediately; plants also store it as starch or convert it into other organic compounds like cellulose, which is essential for building their cellular structure.
122
+
This energy reserve allows them to grow, develop leaves, produce flowers, bear fruit, and carry out various physiological processes throughout their lifecycle."""
-[`LEDForConditionalGeneration`] is an extension of [`BartForConditionalGeneration`] exchanging the traditional self-attention layer with Longformer's chunked self-attention layer. [`LEDTokenizer`] is an alias of [`BartTokenizer`].
136
+
- LED pads the `input_ids` to be a multiple of `config.attention_window` if required. A small speedup is gained when [`LEDTokenizer`] is used with the `pad_to_multiple_of` argument.
137
+
- LED works best on long-range sequence-to-sequence tasks where the `input_ids` are significantly longer than 1024 tokens.
138
+
- LED uses global attention by means of the `global_attention_mask` (see [`LongformerModel`]). For summarization, it is advised to put global attention only on the first `<s>` token. For question answering, it is advised to put global attention on all tokens of the question.
139
+
- To fine-tune LED on all 16384 parameters, gradient checkpointing can be enabled in case training leads to out-of-memory (OOM) errors. Enable gradient checkpointing by adding `model.gradient_checkpointing_enable()` and setting `use_cache=False` to disable the caching mechanism to save memory.
140
+
- Inputs should be padded on the right because LED uses absolute position embeddings.
64
141
65
142
## Resources
66
143
67
-
-[A notebook showing how to evaluate LED](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing).
68
-
-[A notebook showing how to fine-tune LED](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing).
- Read the [LED on Arxiv notebook](https://colab.research.google.com/drive/12INTTR6n64TzS4RrXZxMSXfrOd9Xzamo?usp=sharing) to see how LED can achieve state-of-the-art performance on Arxiv article summarization.
145
+
- Read the [Fine-tune LED notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing) to learn how to fine-tune LED on PubMed articles.
0 commit comments