Repository navigation
Conversation
update_scale only backed off when delayed_shift or cur_hysteresis was exactly 1, so fp16.hysteresis of 0 or less skipped every overflowing step forever. Treat values <= 1 as no delay, as the stage 0 optimizers do. Signed-off-by: Ovidiu Damian <damian.ovidiu27@gmail.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 926c80bdcd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for _ in range(3): | ||
| scaler.update_scale(True) | ||
| assert scaler.loss_scale == 2**5 |
There was a problem hiding this comment.
Add and run a ZeRO regression for nonpositive hysteresis
For fp16 ZeRO runs with hysteresis=0 or a negative value, this coverage only calls scaler.update_scale(True); inheriting DistributedTest does not exercise engine initialization, overflow detection, skipped weight updates, or recovery on a finite step. The overflow integration cases in test_zero_optim_overflow.py hard-code hysteresis 1, and the commit explicitly reports no ZeRO end-to-end run. AGENTS.md requires “integration tests, not just unit tests” for training-loop contract changes and requires execution on actual devices. Add a minimal SimpleModel ZeRO regression for these values, verify scale backoff and a subsequent finite parameter update, and record the hardware and execution results.
AGENTS.md reference: AGENTS.md:L81-L82
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
🟡 Changes recommended
Regression coverage and fp16 accelerator results are still needed to verify that ZeRO reduces the scale, skips overflowing updates, and resumes finite updates with hysteresis 0 and -1.
1 open issue
What changed in this PR
Fixes #8982 by making ZeRO’s dynamic loss scaler treat nonpositive fp16 hysteresis as no delay, matching stage 0 behavior without rejecting existing configurations.
Changes:
- Uses
<= 1for overflow backoff checks, preserving behavior for hysteresis values of 1 or greater. - Adds a parameterized scaler regression for hysteresis 1, 0, and -1.
| File | Description |
|---|---|
| tests/unit/runtime/half_precision/test_dynamic_loss_scale.py | Checks scale reduction after repeated overflows for no-delay settings. |
| deepspeed/runtime/fp16/loss_scaler.py | Allows scale backoff when hysteresis is zero or negative. |
Review effort: Balanced
| if self.delayed_shift == 1 or self.cur_hysteresis == 1: | ||
| # Back off once the hysteresis budget is used up. `<= 1` (not `== 1`) so that a configured | ||
| # hysteresis of 0 or below means "no delay" instead of never lowering the scale. | ||
| if self.delayed_shift <= 1 or self.cur_hysteresis <= 1: |

DeepSpeedFP16Configacceptsfp16.hysteresisof 0 or a negative number.DynamicLossScaler.update_scale, which ZeRO stages 1, 2 and 3 use throughCreateLossScaler, only lowers the scale whendelayed_shift == 1 or cur_hysteresis == 1. With a start value of 0 or below,cur_hysteresisonly counts further down, so the scale is never lowered and every overflowing step is skipped for the rest of the run. The stage 0 fp16 optimizers already treat these values as no delay (they back off unlesscur_hysteresis > 1), so the two paths disagreed.The change uses
<= 1in both comparisons, so 0 or below behaves like 1 and backs off on every overflow. Values of 1 and above behave as before. I did not add config validation, to avoid rejecting configs that load today.Fixes #8982
Duplicate check
The related ones are #8647, which added hysteresis handling to the stage 0 optimizers with a
> 1check but leftDynamicLossScalerunchanged, and the open #7856, which would rejecthysteresisbelow 1 at config parse time. This change takes the other approach and accepts 0 or less as no delay, the same as the stage 0 optimizers, so either can land. Sibling paths:fused_optimizer.pyandunfused_optimizer.py(stage 0) already usecur_hysteresis > 1, and bf16 has no loss scaler. The change touches onlyDynamicLossScaler.Tests
Added
TestZeroScalerHysteresis::test_no_delay_lowers_the_scaleintests/unit/runtime/half_precision/test_dynamic_loss_scale.py. For hysteresis 1, 0 and -1 it sends three overflows and checks the scale goes from 28 to 25.Run in Docker on CPU only (no GPU; x86_64 Linux, Python 3.13, torch 2.9.0+cpu,
DS_ACCELERATOR=cpu,DS_BUILD_OPS=0):pytest tests/unit/runtime/half_precision/test_dynamic_loss_scale.py -k TestZeroScalerHysteresis: 2 failed and 1 passed on main, 3 passed with the fix.Not run: the rest of that test file (it needs an fp16-capable accelerator), pre-commit, ZeRO end-to-end runs.
Compatibility matrix
Only the scaler class was exercised, which is the code ZeRO 1, 2 and 3 share with fp16. Not validated: ZeRO end-to-end runs, offload, parallelism, GPUs. There is no change to numerics, throughput or memory for hysteresis of 1 or more.
This pr was prepared with AI assistance and reviewed by a human.