Issues

• Fix IndexError in get_num_experts_per_layer for dense models under pipeline parallelism

  deepspeedai/Megatron-DeepSpeed#486 · Filter by author hipa211a opened on Jul 16, 2026

• [finetune_hf_llama example] hf2megads_weight_converter.py fails to save due to Attrubute error

  deepspeedai/Megatron-DeepSpeed#482 · Filter by author aravneelaws opened on Sep 24, 2025

• [QUESTION]How to train llama 3 with Megatron-DeepSpeed

  deepspeedai/Megatron-DeepSpeed#481 · Filter by author lzyonetwothree opened on Sep 24, 2025
  4

• [QUESTION] lr schedule(OptimizerParamScheduler) is not aligned when using token and sample as argument

  deepspeedai/Megatron-DeepSpeed#475 · Filter by author xrrain opened on May 6, 2025

• can we [add new feature ]support zero=2,3 with --tensor-model-parallel-size 2 --pipeline-model-parallel-size 2 for pretrain-gpt2?

  deepspeedai/Megatron-DeepSpeed#474 · Filter by author SeekPoint opened on Apr 28, 2025

• [BUG] : training gpt2 with PP=2 then error: list index out of range

  deepspeedai/Megatron-DeepSpeed#473 · Filter by author 9LLPPLL6 opened on Apr 18, 2025

• Tutorials does not reflect contents in repository

  deepspeedai/Megatron-DeepSpeed#471 · Filter by author auggie246 opened on Mar 26, 2025

• support qwen2.5-32b?

  deepspeedai/Megatron-DeepSpeed#466 · Filter by author RyanOvO opened on Feb 24, 2025

• Does it support deployment and invocation under the Ascend system & CANN?

  deepspeedai/Megatron-DeepSpeed#465 · Filter by author RyanOvO opened on Feb 21, 2025

• Converting hf weights to mds: unable to find latest file

  deepspeedai/Megatron-DeepSpeed#464 · Filter by author anasegor opened on Feb 17, 2025

• Finetune issue with Llama2-7b

  deepspeedai/Megatron-DeepSpeed#463 · Filter by author anasegor opened on Feb 14, 2025

• Any one used “—exit-signal-handler” ?

  deepspeedai/Megatron-DeepSpeed#460 · Filter by author HenryTangDev opened on Feb 5, 2025

• Model Inference issues with GPT-MoE models

  deepspeedai/Megatron-DeepSpeed#458 · Filter by author 1155157110 opened on Jan 25, 2025

• Train GPT2 with 2nodes, 2gpu, tp=2 IndexError at megatron/core/tensor_parallel/cross_entropy.py

  deepspeedai/Megatron-DeepSpeed#457 · Filter by author yuanpeng-zhu opened on Jan 13, 2025

• Universal checkpoint for megatron

  deepspeedai/Megatron-DeepSpeed#453 · Filter by author JiashuWu opened on Dec 7, 2024

• Model conversion problem

  deepspeedai/Megatron-DeepSpeed#449 · Filter by author yuanzhiyong1999 opened on Sep 26, 2024
  1

• Async allreduce for tensor-parallel

  deepspeedai/Megatron-DeepSpeed#447 · Filter by author drcanchi opened on Sep 23, 2024
  1

• [TRACKER] Customer support related PR tracker for Intel devices

  deepspeedai/Megatron-DeepSpeed#446 · Filter by author delock opened on Sep 20, 2024

• how to calcuate the training throughput

  deepspeedai/Megatron-DeepSpeed#444 · Filter by author bigtree2020 opened on Sep 12, 2024

• llama3 and llama3.1 support

  deepspeedai/Megatron-DeepSpeed#443 · Filter by author fmiao2372 opened on Sep 10, 2024
  1

• [Bug] Missing weight gradients from LinearWithGradAccumulationAndAsyncCommunication when Zero Bubble Pipeline Parallelism Is disabled

  deepspeedai/Megatron-DeepSpeed#442 · Filter by author mksit opened on Sep 1, 2024

• Optimizer problem when using finetune_llama.sh

  deepspeedai/Megatron-DeepSpeed#440 · Filter by author Kaiizx opened on Aug 28, 2024
  3

• zero3 The checkpoint being loaded used a DP world size of 8 but the current world size is 16. Automatic adjustment of ZeRO's optimizer state partitioning with a new world size is not currently supported.

  deepspeedai/Megatron-DeepSpeed#439 · Filter by author ArtificialZeng opened on Aug 26, 2024

• Why pretrain_llama_distributed.sh use pretrain_gpt.py ?

  deepspeedai/Megatron-DeepSpeed#437 · Filter by author BrucePeng92 opened on Aug 22, 2024

• [bug]: ipex install breaks non xpu devices

  deepspeedai/Megatron-DeepSpeed#435 · Filter by author saforem2 opened on Aug 8, 2024
  2