Issues

• can we [add new feature ]support zero=2,3 with --tensor-model-parallel-size 2 --pipeline-model-parallel-size 2 for pretrain-gpt2?

  bigscience-workshop/Megatron-DeepSpeed#407 · Filter by author SeekPoint opened on Apr 28, 2025
  1

• BUG: training gpt2 with pp=2 error:list index out of range

  bigscience-workshop/Megatron-DeepSpeed#406 · Filter by author 9LLPPLL6 opened on Apr 18, 2025

• How can I set recomputation-granularity,like selective or full?

  bigscience-workshop/Megatron-DeepSpeed#403 · Filter by author LordEdison opened on Apr 30, 2024

• Hello, what version of the megatron-lm library is your code modified?

  bigscience-workshop/Megatron-DeepSpeed#401 · Filter by author 4thGardenOfQMH opened on Feb 26, 2024

• Is this assertion for mask wrong?

  bigscience-workshop/Megatron-DeepSpeed#400 · Filter by author yinfangchen opened on Feb 15, 2024
  1

• Hello, can Megatron-DeepSpeed pre-train llama2?

  bigscience-workshop/Megatron-DeepSpeed#398 · Filter by author 13416157913 opened on Oct 12, 2023
  2

• the traing log like this is Normal？ I do not find loss in the logs, and what does the "grad norm: nan" mean?

  bigscience-workshop/Megatron-DeepSpeed#396 · Filter by author alphanlp opened on Aug 27, 2023

• The difference between zero-3 and megatron with zero-2

  bigscience-workshop/Megatron-DeepSpeed#395 · Filter by author nicosouth opened on Aug 25, 2023

• Question about the implementation of mpu.cross_entropy when using tensor parallel

  bigscience-workshop/Megatron-DeepSpeed#394 · Filter by author robin087 opened on Aug 3, 2023

• questions about inconsistent evaluation result

  bigscience-workshop/Megatron-DeepSpeed#392 · Filter by author coorful opened on Jul 24, 2023
  1

• Question about ds to universal

  bigscience-workshop/Megatron-DeepSpeed#388 · Filter by author saxh opened on May 31, 2023

• RuntimeError: Error building extension 'scaled_upper_triang_masked_softmax_cuda'

  bigscience-workshop/Megatron-DeepSpeed#387 · Filter by author zll0000 opened on May 26, 2023
  1

• hello， I meet a problem

  bigscience-workshop/Megatron-DeepSpeed#386 · Filter by author etoilestar opened on May 22, 2023
  8

• How to properly use Flops Profiler with pipelined parallelism?

  bigscience-workshop/Megatron-DeepSpeed#385 · Filter by author flyingdown opened on May 9, 2023

• pip install -e . failed with ModuleNotFoundError: No module named 'torch'

  bigscience-workshop/Megatron-DeepSpeed#383 · Filter by author SeekPoint opened on May 6, 2023
  2

• Help me, I'm dying soon，error: command '/opt/rh/devtoolset-7/root/usr/bin/gcc' failed with exit code 1 error: subprocess-exited-with-error

  bigscience-workshop/Megatron-DeepSpeed#382 · Filter by author listwebit opened on May 5, 2023

• Megatron-DeepSpeed only applies to specific models?

  bigscience-workshop/Megatron-DeepSpeed#381 · Filter by author Bob-cby opened on May 4, 2023

• The given group does not exist pytorch

  bigscience-workshop/Megatron-DeepSpeed#379 · Filter by author germanjke opened on Apr 25, 2023
  2

• upgrade megatron-lm

  bigscience-workshop/Megatron-DeepSpeed#378 · Filter by author dz1iang opened on Apr 21, 2023

• How can we access to the gradients while the model is training?

  bigscience-workshop/Megatron-DeepSpeed#377 · Filter by author BilgehanSel opened on Apr 19, 2023

• how to do prompt learning with bloom?

  bigscience-workshop/Megatron-DeepSpeed#376 · Filter by author moseshu opened on Apr 10, 2023

• how to frozen some layers of GPT, only fintune last k layers?

  bigscience-workshop/Megatron-DeepSpeed#375 · Filter by author joan126 opened on Apr 4, 2023

• Can I use python only apex for gpt_pretrain?

  bigscience-workshop/Megatron-DeepSpeed#373 · Filter by author Luoyang144 opened on Mar 21, 2023

• how to pretrain t5-lm adapted?

  bigscience-workshop/Megatron-DeepSpeed#372 · Filter by author nanyyyyyy opened on Mar 18, 2023

• How to preprocess data for t5 model?

  bigscience-workshop/Megatron-DeepSpeed#371 · Filter by author xiu-ze opened on Mar 14, 2023