Repository navigation
Fix-FP16-ZeRO-Stage-0-loss-Scaling-In-engine.backward() - #8393
etemadiamd wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 857e98cbee
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| # loss.backward() directly makes effective gradient updates ~cur_scale | ||
| # times too small, stalling training. Route through | ||
| # FP16_Optimizer.backward() which applies the scaling correctly. | ||
| self.optimizer.backward(loss, **backward_kwargs) |
There was a problem hiding this comment.
Preserve graph registration before FP16 backward
When gradient_accumulation_steps > 1, this early backward call and return bypass the graph registration at lines 3341–3344. The output hooks installed by forward() therefore make _backward_prologue_per_tensor() divide each output gradient by the accumulation count, even though the loss was already divided at line 3322, attenuating FP16 ZeRO-0 updates by an additional factor of gradient_accumulation_steps. Keep the scaled backward on the common registered path, or register the graph before invoking the optimizer.
Useful? React with 👍 / 👎.
| # TODO: handle these scaling with direct calls to loss.backward() | ||
| if isinstance(self.optimizer, ZeROOptimizer): | ||
| loss = self.optimizer.scale_if_loss(loss) | ||
| elif self.fp16_enabled() and not self.zero_optimization(): |
There was a problem hiding this comment.
Add the mandatory sign-off trailer
This is a single-parent, non-merge commit, but its commit message has no Signed-off-by trailer, so it does not meet the repository's mandatory commit requirement and should be recreated with --signoff.
AGENTS.md reference: AGENTS.md:L8-L8
Useful? React with 👍 / 👎.
| self.optimizer.backward(loss, **backward_kwargs) | ||
| self._backward_epilogue() | ||
| self._running_engine_backward = False | ||
| return gas_scaled_loss |
There was a problem hiding this comment.
Keep FP16 backward inside the compiled-autograd context
When callers enable compiled_autograd_enabled=True through DeepSpeedEngine.compile(), every other supported backward path executes inside compiled_autograd(...), but this FP16 ZeRO-0 branch performs the actual backward and returns before entering that context. Training remains eager despite the explicitly enabled option, losing the expected compiled-backward behavior and performance for this configuration.
Useful? React with 👍 / 👎.
857e98c to
e614064
Compare
…i#8647) At ZeRO stage 0 (fp16 with no `zero_optimization` block, or `"stage": 0`), `fp16.hysteresis` and `fp16.consecutive_hysteresis` are silently ignored. The loss scale halves on every overflow. The same config at ZeRO 1, 2 or 3 goes through `DynamicLossScaler` and honors both. `LossScaleConfig` copies the initial scale, window and minimum out of `dynamic_loss_scale_args()` and drops `delayed_shift` and `consecutive_hysteresis`. `FP16_Optimizer._update_scale` and `FP16_UnfusedOptimizer._update_scale` then have no hysteresis logic at all. This carries both settings through and applies them the way `DynamicLossScaler.update_scale` does. It also saves the remaining hysteresis in the stage 0 state dict (with a fallback for older checkpoints), and stops the overflow log from saying the scale was reduced when hysteresis absorbed it. Four overflows with `hysteresis=3`, `initial_scale_power=8`: ``` before [128, 64, 32, 16] after [256, 256, 128, 64] (same as ZeRO 1/2/3) ``` This changes the default behavior at stage 0: the default `hysteresis=2` now takes effect, which matches ZeRO and the docs. The four existing stage 0 overflow tests relied on the default being ignored, so they now pin `"hysteresis": 1`, the way the ZeRO overflow tests already do. The separate open deepspeedai#8393 (stage 0 fp16 backward not scaling the loss) is on the same path but is a different bug. Tests: new `TestZeroStage0Hysteresis` in `test_dynamic_loss_scale.py`, over both optimizers, with three cases: hysteresis, consecutive hysteresis, and a checkpoint round trip. 6 pass here. On master, a single-process run of the same checks gives the "before" row above. yapf 0.43.0 and flake8 5.0.4 are clean. Executed on Apple M2 Pro CPU. The GPU-gated stage 0 overflow tests could not run here. --------- Signed-off-by: Vineeth Sai <vineethsai4444@gmail.com> Co-authored-by: pengdurice <pengduhit@gmail.com>
When fp16 is enabled with ZeRO stage 0, engine.backward() falls through to loss.backward() directly. FP16_Optimizer.step() always divides gradients by cur_scale (typically 32768), so gradients that were never pre-scaled are unscaled by the same factor, making effective parameter updates ~cur_scale times too small and stalling training. Add an explicit elif branch for fp16 + ZeRO-0 that routes through FP16_Optimizer.backward(), which pre-scales the loss by cur_scale before calling backward(), matching what step() expects. The existing # TODO comment on line 3325 acknowledges this path was left incomplete. The ZeRO stages > 0 and AMP paths are handled correctly; only the fp16 non-ZeRO path was missing. Tested with a CIFAR-10 MoE benchmark (fp16, ZeRO stage 0, 8 experts): - Before: accuracy 22-28%, loss stalls at ~1.8 - After: accuracy 49%, loss converges to ~0.7 Signed-off-by: Roohollah Etemadi <Roohollah.Etemadi@amd.com>
The early return in engine.backward() skipped the loss hook that marks the graph as loss-scaled, so output backward hooks divided grads by gradient_accumulation_steps a second time. Dispatch to FP16_Optimizer.backward() from the common path instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Roohollah Etemadi <Roohollah.Etemadi@amd.com>
4aa8b2d to
443186d
Compare
When fp16 is enabled with ZeRO stage 0, engine.backward() falls through to loss.backward() directly. FP16_Optimizer.step() always divides gradients by cur_scale (typically 32768), so gradients that were never pre-scaled are unscaled by the same factor, making effective parameter updates ~cur_scale times too small and stalling training.
Add an explicit elif branch for fp16 + ZeRO-0 that routes through FP16_Optimizer.backward(), which pre-scales the loss by cur_scale before calling backward(), matching what step() expects.
The existing # TODO comment on line 3325 acknowledges this path was left incomplete. The ZeRO stages > 0 and AMP paths are handled correctly; only the fp16 non-ZeRO path was missing.
Tested with a CIFAR-10 MoE benchmark (fp16, ZeRO stage 0, 8 experts):