Add support for MiniMax's MiniMax-Text-01

[geetu040]

What does this PR do?

Fixes #35710

This PR adds MiniMaxAI's MiniMax-Text-01 model to Hugging Face Transformers.

• MiniMax-Text-01 is a powerful language model with 456 billion total parameters, of which 45.9 billion are activated per token.
• To better unlock the long context capabilities of the model, MiniMax-Text-01 adopts a hybrid architecture that combines Lightning Attention, Softmax Attention and Mixture-of-Experts (MoE).
• MiniMax-Text-01's training context length is extended to 1 million tokens, and it can handle a context of up to 4 million tokens during the inference.
• On various academic benchmarks, MiniMax-Text-01 also demonstrates the performance of a top-tier model.

Relevant Links

• Research Paper: MiniMax-01: Scaling Foundation Models with Lightning Attention
• Authors: MiniMax, Aonian Li, Bangwei Gong, et al.
• Implementation: MiniMaxAI/MiniMax-Text-01
• Models Weights: MiniMaxAI/MiniMax-Text-01
• Github Page: MiniMax-AI/MiniMax-01

CC: @MiniMax-AI-Dev

Before submitting

• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@ArthurZucker, @Rocketknight1

Change Log

• Tokenizer: It uses the existing GPT2Tokenizer
• Config: Matches the MixtralConfig with a few additional parameters:
  • residual_post_norm, attn_type_list
  • layernorm_attention_alpha, layernorm_lightning_attention_alpha, layernorm_mlp_alpha
  • layernorm_attention_beta, layernorm_lightning_attention_beta, layernorm_mlp_beta
• Weight Conversion Script: No script needed, original weights can be loaded directly into the new architecutre
• Model: MiniMax-Text-01 architecture matches and uses most of the Mixtral architecture, with a few changes in
  • DecoderLayer
    • hidden_states can be used as residual connections, before or after layernorm is applied
    • weighted sum is used in residual connection
    • selection between softmax and lightning attention based on the layer_idx
  • LightningAttention
    • intially used in TransNormerLLM, upgraded in Lightning Attention-2 and adopted in MiniMax-01
    • every 8th decoder layer uses a softmax attention, rest of the layers use lightning attention, which is not previously implemented in transformers

To summarize above, the main area of review is the LightningAttention implementation.

TODOs

• Update Documentation
• Update Tests
• Import Statements and Auto Modeling
• Implement Model
  • Implement End-to-End Architecture
  • Implement LightningAttention
    • Work with avialable code
    • Refactor, Clean and Optimize
    • Implement Caching
    • Support .generate() method

[Rocketknight1]

Hi @geetu040, this looks quite good! You can ping us whenever it's ready for review. Also, code quality issues in the CI can be fixed with pip install -e .[quality] in the transformers directory, followed by make fixup.