Skip to content

Fix-FP16-ZeRO-Stage-0-loss-Scaling-In-engine.backward() - #8393

Open
etemadiamd wants to merge 2 commits into
deepspeedai:masterfrom
etemadiamd:fix/fp16_zero0_loss_scaling_engine_backward
Open

etemadiamd wants to merge 2 commits into
deepspeedai:masterfrom
etemadiamd:fix/fp16_zero0_loss_scaling_engine_backward

Conversation

@etemadiamd

@etemadiamd etemadiamd commented Sep 2, 2026 •

Copy link
Copy Markdown

When fp16 is enabled with ZeRO stage 0, engine.backward() falls through to loss.backward() directly. FP16_Optimizer.step() always divides gradients by cur_scale (typically 32768), so gradients that were never pre-scaled are unscaled by the same factor, making effective parameter updates ~cur_scale times too small and stalling training.

Add an explicit elif branch for fp16 + ZeRO-0 that routes through FP16_Optimizer.backward(), which pre-scales the loss by cur_scale before calling backward(), matching what step() expects.

The existing # TODO comment on line 3325 acknowledges this path was left incomplete. The ZeRO stages > 0 and AMP paths are handled correctly; only the fp16 non-ZeRO path was missing.

Tested with a CIFAR-10 MoE benchmark (fp16, ZeRO stage 0, 8 experts):

  • Before: accuracy 22-28%, loss stalls at ~1.8
  • After: accuracy 49%, loss converges to ~0.7

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 857e98cbee

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread deepspeed/runtime/engine.py Outdated
# loss.backward() directly makes effective gradient updates ~cur_scale
# times too small, stalling training. Route through
# FP16_Optimizer.backward() which applies the scaling correctly.
self.optimizer.backward(loss, **backward_kwargs)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve graph registration before FP16 backward

When gradient_accumulation_steps > 1, this early backward call and return bypass the graph registration at lines 3341–3344. The output hooks installed by forward() therefore make _backward_prologue_per_tensor() divide each output gradient by the accumulation count, even though the loss was already divided at line 3322, attenuating FP16 ZeRO-0 updates by an additional factor of gradient_accumulation_steps. Keep the scaled backward on the common registered path, or register the graph before invoking the optimizer.

Useful? React with 👍 / 👎.

# TODO: handle these scaling with direct calls to loss.backward()
if isinstance(self.optimizer, ZeROOptimizer):
loss = self.optimizer.scale_if_loss(loss)
elif self.fp16_enabled() and not self.zero_optimization():

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add the mandatory sign-off trailer

This is a single-parent, non-merge commit, but its commit message has no Signed-off-by trailer, so it does not meet the repository's mandatory commit requirement and should be recreated with --signoff.

AGENTS.md reference: AGENTS.md:L8-L8

Useful? React with 👍 / 👎.

Comment thread deepspeed/runtime/engine.py Outdated
Comment on lines +3334 to +3337
self.optimizer.backward(loss, **backward_kwargs)
self._backward_epilogue()
self._running_engine_backward = False
return gas_scaled_loss

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep FP16 backward inside the compiled-autograd context

When callers enable compiled_autograd_enabled=True through DeepSpeedEngine.compile(), every other supported backward path executes inside compiled_autograd(...), but this FP16 ZeRO-0 branch performs the actual backward and returns before entering that context. Training remains eager despite the explicitly enabled option, losing the expected compiled-backward behavior and performance for this configuration.

Useful? React with 👍 / 👎.

@etemadiamd
etemadiamd force-pushed the fix/fp16_zero0_loss_scaling_engine_backward branch from 857e98c to e614064 Compare September 2, 2026 18:45
banxingmjj pushed a commit to openanolis/DeepSpeed that referenced this pull request Oct 27, 2026
…i#8647)

At ZeRO stage 0 (fp16 with no `zero_optimization` block, or `"stage":
0`), `fp16.hysteresis` and `fp16.consecutive_hysteresis` are silently
ignored. The loss scale halves on every overflow. The same config at
ZeRO 1, 2 or 3 goes through `DynamicLossScaler` and honors both.

`LossScaleConfig` copies the initial scale, window and minimum out of
`dynamic_loss_scale_args()` and drops `delayed_shift` and
`consecutive_hysteresis`. `FP16_Optimizer._update_scale` and
`FP16_UnfusedOptimizer._update_scale` then have no hysteresis logic at
all. This carries both settings through and applies them the way
`DynamicLossScaler.update_scale` does. It also saves the remaining
hysteresis in the stage 0 state dict (with a fallback for older
checkpoints), and stops the overflow log from saying the scale was
reduced when hysteresis absorbed it.

Four overflows with `hysteresis=3`, `initial_scale_power=8`:

```
before  [128, 64, 32, 16]
after   [256, 256, 128, 64]    (same as ZeRO 1/2/3)
```

This changes the default behavior at stage 0: the default `hysteresis=2`
now takes effect, which matches ZeRO and the docs. The four existing
stage 0 overflow tests relied on the default being ignored, so they now
pin `"hysteresis": 1`, the way the ZeRO overflow tests already do. The
separate open deepspeedai#8393 (stage 0 fp16 backward not scaling the loss) is on
the same path but is a different bug.

Tests: new `TestZeroStage0Hysteresis` in `test_dynamic_loss_scale.py`,
over both optimizers, with three cases: hysteresis, consecutive
hysteresis, and a checkpoint round trip. 6 pass here. On master, a
single-process run of the same checks gives the "before" row above. yapf
0.43.0 and flake8 5.0.4 are clean. Executed on Apple M2 Pro CPU. The
GPU-gated stage 0 overflow tests could not run here.

---------

Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com>
Co-authored-by: pengdurice <pengduhit@gmail.com>
etemadiamd and others added 2 commits November 10, 2026 19:01
When fp16 is enabled with ZeRO stage 0, engine.backward() falls through
to loss.backward() directly. FP16_Optimizer.step() always divides
gradients by cur_scale (typically 32768), so gradients that were never
pre-scaled are unscaled by the same factor, making effective parameter
updates ~cur_scale times too small and stalling training.

Add an explicit elif branch for fp16 + ZeRO-0 that routes through
FP16_Optimizer.backward(), which pre-scales the loss by cur_scale before
calling backward(), matching what step() expects.

The existing # TODO comment on line 3325 acknowledges this path was
left incomplete. The ZeRO stages > 0 and AMP paths are handled correctly;
only the fp16 non-ZeRO path was missing.

Tested with a CIFAR-10 MoE benchmark (fp16, ZeRO stage 0, 8 experts):
- Before: accuracy 22-28%, loss stalls at ~1.8
- After:  accuracy 49%, loss converges to ~0.7

Signed-off-by: Roohollah Etemadi <Roohollah.Etemadi@amd.com>
The early return in engine.backward() skipped the loss hook that marks the
graph as loss-scaled, so output backward hooks divided grads by
gradient_accumulation_steps a second time. Dispatch to
FP16_Optimizer.backward() from the common path instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Roohollah Etemadi <Roohollah.Etemadi@amd.com>
@etemadiamd
etemadiamd force-pushed the fix/fp16_zero0_loss_scaling_engine_backward branch from 4aa8b2d to 443186d Compare November 10, 2026 19:07

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants