![[D] Interesting Gradient Norm Goes Down-Up-Down](https://www.digitado.com.br/wp-content/uploads/2026/02/hg2fed5u2ejg1-4TXtBI-978x400.png "[D] Interesting Gradient Norm Goes Down-Up-Down")
[D] Interesting Gradient Norm Goes Down-Up-Down
![[D] Interesting Gradient Norm Goes Down-Up-Down](https://preview.redd.it/hg2fed5u2ejg1.png?width=140&height=121&auto=webp&s=c61e1c948658db8d5b030e0293726d132eddbee5 "[D] Interesting Gradient Norm Goes Down-Up-Down") |
When I’m training an MoE model with modelscope-swift (with megatron as the backend), I find the gradient norm goes up and down during the training phase. Although the language modeling loss continually goes down, I want to figure out why the training process would behave like this. Is it a problem, and how to resolve this issue? Some details:
submitted by /u/Spico197 |
Like
0
Liked
Liked
