[docs] Translation guide

[stevhliu]

Fixes #32533 with a clarification.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[amyeroberts]

Thanks!

[frereit]

This is definitely still incorrect. "text_targets" is set to the list of target strings. At no point is the tokenizer even "told" that the target language is french. The text_targets parameter only specifies that this text should appear under the labels key in the returned dictionary. In the code, source_lang and target_lang is only used to grab the correct values from the dataset for inputs and targets, its not passed to the constructor of the tokenizer, or the tokenizer call itself.

To really demonstrate this, simply remkove the prefix from the inputs and observe that the tokenizer tokenizes the "inputs" and "targets" to the EXACT SAME tokens because it is the same, language-agnostic, tokenzier:

source_lang = "en"
target_lang = "fr"
prefix = "translate English to French: "


def preprocess_function(examples):
    # Removing "prefix" to make "inputs" and "targets" equivalent
    inputs = [example[source_lang] for example in examples["translation"]]
    targets = [example[target_lang] for example in examples["translation"]]
    print(f"Passing {inputs=} and {targets=} to the tokenizer.")
    model_inputs = tokenizer(inputs, text_target=targets, max_length=128, truncation=True)
    return model_inputs

tokens = preprocess_function(
    {"translation": [{"en": "This text is in English. If the docs were correct, this would be tokenized differently in the input and targets, but this is not the case.",
                      "fr": "This text is in English. If the docs were correct, this would be tokenized differently in the input and targets, but this is not the case."}]
    })
tokens["input_ids"] == tokens["labels"]

Passing inputs=['This text is in English. If the docs were correct, this would be tokenized differently in the input and targets, but this is not the case.'] and targets=['This text is in English. If the docs were correct, this would be tokenized differently in the input and targets, but this is not the case.'] to the tokenizer.

True

In terms of the model, how would it even work to have a different tokenizer for inputs and outputs? During generation, the output is fed back into the model as far as I know, so how would the model differentiate between "input tokens" and "output tokens that have been fed back as input tokens" in that case?

[amyeroberts]

Hi @frereit - thanks for the detailed explanation! Would you like to open a PR with what you think should be in the docs?

[frereit]

Hi @amyeroberts ,

I would if I could. Unfortunately, I don't think I really have enough understanding of how this all works to be able to do that, as I'm currently just starting out with the huggingface library and LLMs. I stumbled over this while trying to follow the tutorial and observed what I described above, but I'm not confident that I know how to correct the docs without making it worse.

My current understanding is that the tokenizer is just "universal" / used for all languages, but I assume there was a reason why the docs specified that there should be a different tokenizer used for the source and target languages, but I'm not sure how to implement this.