Refactor embedding input/output getter/setter

[molbap]

What does this PR do?

TL;DR

PreTrainedModel now mixes in EmbeddingAccessMixin providing default
get_input_embeddings / set_input_embeddings / get_output_embeddings / set_output_embeddings for all models. These methods should be removed from the codebase unless exceptionally weird.

Details

Uses a new attribute __input_embed_layer = "embed_tokens" by default, and one can change it to set which layer the embeddings should be gotten/set to. Then, assuming embed_tokens is that layer for instance, resolution order is

• self.model.embed_tokens
• self.embed_tokens
• delegate once via base_model_prefix
• else raise (model must override)

get_output_embeddings now auto-returns lm_head only if input embeddings
resolve (so pure audio/vision backbones still return None).

What you usually have to do: nothing.

Override only if:

• embeddings live under a different attribute
• multiple embedding tables
• you need to hide an lm_head (override get_output_embeddings retunring None)
• helper head without token embeddings but still wants tying (override and return the head)

Potential breakages:

• Exotic layouts with custom embedding attr names (must override)

cc @vasqu @zucchini-nlp, minor but to remember for composite models too

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

Boom! unbloating at its best! let's get htis merged!

[molbap]

Down to a few tests failing because of audio embeddings exceptions mostly, solving, then adding set_ and get_ encoder methods and merging this

[ArthurZucker]

small nit but good to go otherwise!

[ArthurZucker]

self.lm_head is the output embedding no?

[molbap]

well the answer is annoying 😬 the input embeddings are defined as the conv2d patch layer

def get_input_embeddings(self):
    return self.embeddings.patch_embeddings

so if get_output_embeddings stays defined, since in the default config this param defaults to True

tie_word_embeddings (`bool`, *optional*, defaults to `True`):

then an attempt at tying the weights is always done at init

        if getattr(self.config.get_text_config(decoder=True), "tie_word_embeddings", True):
            output_embeddings = self.get_output_embeddings()
            if output_embeddings is not None:
                self._tie_or_clone_weights(output_embeddings, self.get_input_embeddings())

which will fail because conv2 patches have no weight attribute. WDYT of reworking/removing this part from tie_weights?
simpler solution is enforcing return None which is what I have done here (and same for the other comment)

[ArthurZucker]

None works, needs a comment!

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: arcee, aria, aya_vision, bamba, bark, bart, beit, bigbird_pegasus, biogpt, bitnet, blenderbot, blenderbot_small, bloom, chameleon, clvp, codegen

[molbap]

run-slow: beit, bark, bart, clvp, glm4, glm4_moe, zamba2

[github-actions]

This comment contains run-slow, running the specified jobs:

models: ['models/bark', 'models/bart', 'models/beit', 'models/clvp', 'models/glm4', 'models/glm4_moe', 'models/zamba2']
quantizations: [] ...

[molbap]

Tested on a few slow models, failings are identical to main. Merging and keeping an eye out.