[DeepSeek-V3] Different rotary embedding implementation between DeepSeek-AI and Transformers

[wwwjn]

Counterpart issue : deepseek-ai/DeepSeek-V3#938

Describe the issue
Hi team, I'm working on reproducing the great deepseek-v3 model on torchtitan . While I'm trying to run numerical verification, I noticed the rotary embedding in HF and this repo is different.

HF: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py#L339

• In HF rotary embedding implementation, they explicitly permute (interleave the odd column and even columns) the q_pe / k_pe.

Deepseek-AI: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py#L339

• In DeepSeek-AI's implementation, they didn't permute the q_pe / k_pe.
• In the convert.py, the script load HF weights as it's original order, and this script didn't permute the weights, to accommodate the ordering difference in apply_rotary_embedding().

And this discrepancy will result in different mathematical results after the attention module of the first dense layer.

I want to double-check with the team if I missed something here. Thank you for your help in advance! cc @tianyu-l

To Reproduce
Environment: transformers ==4.54.0

I run following 2 runs with code: https://github.com/wwwjn/DeepSeek-V3 . I randomized the same inputs for both runs.

1. a single forward pass using HuggingFace transformers, with weights from:https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/tree/main .
   python hf_implementation/hf_implementation.py --num_layers 5 > hf_outputs.txt 2>&1

2. Using this repo's model implementation and run a single forward pass (To simplify, I didn't use the - First, I used the convert.py to convert HF checkpoint weights.
   python convert.py --hf-ckpt-path /data/users/jianiw/dsv3-weights/ --save-path /data/users/jianiw/dsv3-weights-5-layer/ --n-experts 256 --model-parallel 8

   • Second, I run a single forward pass using
     torchrun --nnodes 1 --nproc-per-node 8 inference/run_single_forward.py --config inference/configs/config_671B.json > dsv3-output.txt 2>&1

Here's the detailed numerical comparison I've seen:

Expected behavior
Expected behavior: After first dense layer's attention layer, the output should be almost the same (There might be slightly difference because of fp8 vs other pericision).

Additional context
We observed the same issue for llama3 model before #30872, and get better understanding with @ArthurZucker 's help - The weights for llama3 on HuggingFace is permuted compared to original Meta's weights . so we need to manually permute the weights back to accommodate the rotary embedding implementation difference. Reference: pytorch/torchtitan#335, pytorch/torchtitan#1291 (comment)

[ArthurZucker]

Hey! You had me worried for a while there!
We have this https://github.com/huggingface/transformers/blob/main/src/transformers/models/deepseek_v3/modeling_deepseek_v3.py#L401-L402

which we added exactly for that!
The documentation should probably be updated and if it is not a default, that should be the default!

[wwwjn]

Hey @ArthurZucker Thank so much for your reply!

Let use following terminology to different 2 rotary embedding implementation:

• Implementation No.1: apply_rotary_embedding_interleaved() (Code) : Permute q_pe and k_pe.
• Implementation No.2: apply_rotary_embedding() (code): No permutation of q_pe, k_pe. This rotary embedding is used in DeepSeek-AI github repo.

Implementation No.1 and No.2 are mathematically different given the same k_pe/q_pe inputs, and should correspond to different wq_a / wq_b weights to achieve the same output result.

In my test, I usedconfig = AutoConfig.from_pretrained(local_hf_path, trust_remote_code=True); model = AutoModelForCausalLM.from_pretrained(local_hf_path, config=config, yyy). transformers leads to the same function(link) as Implementation No.1.

I have few follow up questions:

1. In reality, transformers will go to if self.config.rope_interleave branch and call implementation No.1, and rope_interleave default should be True. Is this correct understanding? I checked the configuration here

2. Is the checkpoint weights from HF repo corresponding to Implementation No.1 or Implementation No.2? If it corresponds to implementation No.1, is there any way I could permute back the weights to make it work with implementation No.2 (Deepseek-AI original implementation)?

Much appreciated!

[wwwjn]

Hey @ArthurZucker , thanks for your help and any updates here? :)

[ArthurZucker]

Hey! Sorry forgot to answer

[ArthurZucker]

1. Deepseek uses the complex formulation, which used to be licenced by meta, which is why transformers went with Eleuther's equivalent implementation that does permute the q and k. BUT this is a bit less efficient, so for most of the recent models (like llama) we permuted q and k in the weights, this way we have a more efficient (but equivalently efficient to the complex formulation) implementation.
2. The original weights can be permutted for your needs of course. We have no power on the organization's checkpoints, my recommendation is to use a branch (you open a PR to that repo with permuted weights) and use it!

[ArthurZucker]

I am sorry for the confusion it created and very happy if you want to submit a PR to update our doc about that!

[ArthurZucker]

TLDR:

• if you use rotated weights, you don't need the permute
• deepseek original codebase does not need the permute because they use the complex formulation
• if you use original weights, but transformers eleuther based implementation of rope, you need to permute the weights or permute q, k on the fly (as it is done today, which is not super efficient 😉 )

[wwwjn]

@ArthurZucker Thanks! We don't have access to deepseek's original weights as well , and we only have access to the weights on HF.

So, to simplify, we should use one of the following options:

1. rotated weights <-> Don't permute q, k in RoPE (deepseek-AI codebase)
2. non-rotated weights <-> permute q, k in RoPE (HF transofmers codebase)

So the deepseek-v3's weights on HF (here) has NOT been rotated, is this a correct understanding?

[wwwjn]

deepseek original codebase does not need the permute because they use the complex formulation

Also have another question about this conclusion. I think the permutation is not cause by complex formulation or Eleuther's formulation. Both formulation are equivalent if they take the same q, k input.

The q, k need to be permute is because the wq / wk that used to generate q, k is different (the columns has been rearranged).

I also want to check if my understanding about this conclusion is correct.

[wwwjn]

Hi @ArthurZucker great Monday! Thank you so much for your constantly help and reply, I want to quickly follow up and double confirm this question

@ArthurZucker Thanks! We don't have access to deepseek's original weights as well , and we only have access to the weights on HF.

So, to simplify, we should use one of the following options:

1. rotated weights <-> Don't permute q, k in RoPE (deepseek-AI codebase)
2. non-rotated weights <-> permute q, k in RoPE (HF transofmers codebase)

So the deepseek-v3's weights on HF (here) has NOT been rotated, is this a correct understanding?

[ArthurZucker]

Yes sorry!

We don't have access to deepseek's original weights as well , and we only have access to the weights on HF.

The original weights, AFAIK are the ones published on huggingface no?

So, to simplify, we should use one of the following options:
rotated weights <-> Don't permute q, k in RoPE (deepseek-AI codebase)
non-rotated weights <-> permute q, k in RoPE (HF transofmers codebase)

Absolutely! The only thing is https://huggingface.co/deepseek-ai/DeepSeek-R1/blob/main/modeling_deepseek.py#L364 the original code DOES permute! On the fly at that!

Regarding:

Also have another question about this conclusion. I think the permutation is not cause by complex formulation or Eleuther's formulation. Both formulation are equivalent if they take the same q, k input.

The q, k need to be permute is because the wq / wk that used to generate q, k is different (the columns has been rearranged).

Yes, but we HAVE to re-arrange them in the weights, otherwise we need to do it in the forward to have correct outputs.

[ArthurZucker]

TLDR of the entire issue:

• rotated weights <-> Eleuther formulation -> Don't permute q, k in RoPE (Llama in transformers)
• non rotated-weights <-> Llama licensed formulation (complex) -> Don't permute in ROPE (deepseek original codebase)
• non-rotated weights <-> Eleuther formulation -> permute q, k in RoPE (Deepseek remote code codebase)

[ArthurZucker]

@wwwjn mega sorry for answering slowly, it's a very important issue but I am just completely under water!

[wwwjn]

Thank you so much for the help and clarification! That helps a lot!!

I tried to write a minimal reproduce of what I observed before: https://gist.github.com/wwwjn/381771de10d1a30b9f7ca2df433e04a3. The script is comparing the following 2 paths:

1. transformers and Deepseek remote code codebase:
   • non-rotated weights <-> Eleuther formulation -> permute q, k in RoPE (Deepseek remote code codebase)
2. Deepseek original codebase:
   • non rotated-weights <-> Llama licensed formulation (complex) -> Don't permute in ROPE

By running the script, I could see these 2 implementation are mathematical NOT equal with some dimension (eg, seq_len=2048, num_heads=128, rope_head_dim=64). Could you help quickly checking if this is expected?

[ArthurZucker]

Yes!
Implementations produce slightly different results. this is absolutely expected!

This does not induce an error that propagates, so it's safe.
One thing, when there is a layer norm after the rope, it can get messy (depending on the type, layer-wise etc you need to shift the distribution of the weights as well to match the inputs q and k that were shifted)

Do you think we can close this issue?