Skip to content

Commit

Permalink
[docs] Translation guide (#32547)
Browse files Browse the repository at this point in the history
clarify
  • Loading branch information
stevhliu authored Aug 8, 2024
1 parent 54ac39c commit 85817d9
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/source/en/tasks/translation.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ The next step is to load a T5 tokenizer to process the English-French language p
The preprocessing function you want to create needs to:

1. Prefix the input with a prompt so T5 knows this is a translation task. Some models capable of multiple NLP tasks require prompting for specific tasks.
2. Tokenize the input (English) and target (French) separately because you can't tokenize French text with a tokenizer pretrained on an English vocabulary.
2. Set the target language (French) in the `text_target` parameter to ensure the tokenizer processes the target text correctly. If you don't set `text_target`, the tokenizer processes the target text as English.
3. Truncate sequences to be no longer than the maximum length set by the `max_length` parameter.

```py
Expand Down

0 comments on commit 85817d9

Please sign in to comment.