[VERY WIP] DSV3

[SalmanMohammadi]

TODO

• Verify parity with https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py using moonshotai/Moonlight-16B-A3B
• Verify RoPE + add YaRN
• Add DSV3 tokenizer
• Training run with moonshotai/Moonlight-16B-A3B
• Training run with DSV3
• Documentation
• Tests

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/2764

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 2 Cancelled Jobs

As of commit f785816 with merge base 0d90675 ():

NEW FAILURES - The following jobs have failed:

• GPU tests / gpu_test (3.11, stable) (gh)
  tests/torchtune/modules/test_attention_utils.py::TestSDPAOrFlexAttention::test_flex_attention
• Lint / lint (3.10) (gh)
  torchtune/models/deepseek_v3/_position_embeddings.py:6:1: W391 blank line at end of file

CANCELLED JOBS - The following jobs were cancelled. Please retry:

• GPU tests / gpu_test (3.10, stable) (gh)
  tests/torchtune/modules/test_attention_utils.py::TestSDPAOrFlexAttention::test_flex_attention
• GPU tests / gpu_test (3.9, stable) (gh)
  tests/torchtune/modules/test_attention_utils.py::TestSDPAOrFlexAttention::test_flex_attention

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[krammnic]

@SalmanMohammadi Magnificent PR! DSV3 tokenizer seemed to me little bit tricky + it will require extra unit test. We can try to delegate it on #2723