Skip to content

Fix DynamicLossScaler never lowering the scale when fp16 hysteresis is 0 or less - #8983

Open
damienen wants to merge 1 commit into
deepspeedai:masterfrom
damienen:fix/deepspeed-01
Open

damienen wants to merge 1 commit into
deepspeedai:masterfrom
damienen:fix/deepspeed-01

Conversation

@damienen

@damienen damienen commented Nov 2, 2026

Copy link
Copy Markdown

DeepSpeedFP16Config accepts fp16.hysteresis of 0 or a negative number. DynamicLossScaler.update_scale, which ZeRO stages 1, 2 and 3 use through CreateLossScaler, only lowers the scale when delayed_shift == 1 or cur_hysteresis == 1. With a start value of 0 or below, cur_hysteresis only counts further down, so the scale is never lowered and every overflowing step is skipped for the rest of the run. The stage 0 fp16 optimizers already treat these values as no delay (they back off unless cur_hysteresis > 1), so the two paths disagreed.

The change uses <= 1 in both comparisons, so 0 or below behaves like 1 and backs off on every overflow. Values of 1 and above behave as before. I did not add config validation, to avoid rejecting configs that load today.

Fixes #8982

Duplicate check

The related ones are #8647, which added hysteresis handling to the stage 0 optimizers with a > 1 check but left DynamicLossScaler unchanged, and the open #7856, which would reject hysteresis below 1 at config parse time. This change takes the other approach and accepts 0 or less as no delay, the same as the stage 0 optimizers, so either can land. Sibling paths: fused_optimizer.py and unfused_optimizer.py (stage 0) already use cur_hysteresis > 1, and bf16 has no loss scaler. The change touches only DynamicLossScaler.

Tests

Added TestZeroScalerHysteresis::test_no_delay_lowers_the_scale in tests/unit/runtime/half_precision/test_dynamic_loss_scale.py. For hysteresis 1, 0 and -1 it sends three overflows and checks the scale goes from 28 to 25.

Run in Docker on CPU only (no GPU; x86_64 Linux, Python 3.13, torch 2.9.0+cpu, DS_ACCELERATOR=cpu, DS_BUILD_OPS=0):

  • pytest tests/unit/runtime/half_precision/test_dynamic_loss_scale.py -k TestZeroScalerHysteresis: 2 failed and 1 passed on main, 3 passed with the fix.
  • Standalone script from the issue on main: hysteresis 0 and -1 keep scale 65536 after 20 overflows (exit 1). With the fix the scale reaches the minimum (exit 0).

Not run: the rest of that test file (it needs an fp16-capable accelerator), pre-commit, ZeRO end-to-end runs.

Compatibility matrix

Only the scaler class was exercised, which is the code ZeRO 1, 2 and 3 share with fp16. Not validated: ZeRO end-to-end runs, offload, parallelism, GPUs. There is no change to numerics, throughput or memory for hysteresis of 1 or more.

This pr was prepared with AI assistance and reviewed by a human.

update_scale only backed off when delayed_shift or cur_hysteresis was
exactly 1, so fp16.hysteresis of 0 or less skipped every overflowing
step forever. Treat values <= 1 as no delay, as the stage 0 optimizers do.

Signed-off-by: Ovidiu Damian <damian.ovidiu27@gmail.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-02T12:46:29.573439Z 926c80b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 926c80bdcd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +394 to +396
for _ in range(3):
scaler.update_scale(True)
assert scaler.loss_scale == 2**5

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add and run a ZeRO regression for nonpositive hysteresis

For fp16 ZeRO runs with hysteresis=0 or a negative value, this coverage only calls scaler.update_scale(True); inheriting DistributedTest does not exercise engine initialization, overflow detection, skipped weight updates, or recovery on a finite step. The overflow integration cases in test_zero_optim_overflow.py hard-code hysteresis 1, and the commit explicitly reports no ZeRO end-to-end run. AGENTS.md requires “integration tests, not just unit tests” for training-loop contract changes and requires execution on actual devices. Add a minimal SimpleModel ZeRO regression for these values, verify scale backoff and a subsequent finite parameter update, and record the hardware and execution results.

AGENTS.md reference: AGENTS.md:L81-L82

Useful? React with 👍 / 👎.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Regression coverage and fp16 accelerator results are still needed to verify that ZeRO reduces the scale, skips overflowing updates, and resumes finite updates with hysteresis 0 and -1.

1 open issue
What changed in this PR

Fixes #8982 by making ZeRO’s dynamic loss scaler treat nonpositive fp16 hysteresis as no delay, matching stage 0 behavior without rejecting existing configurations.

Changes:

  • Uses <= 1 for overflow backoff checks, preserving behavior for hysteresis values of 1 or greater.
  • Adds a parameterized scaler regression for hysteresis 1, 0, and -1.
File Description
tests/​unit/​runtime/​half_precision/​test_dynamic_loss_scale.py Checks scale reduction after repeated overflows for no-delay settings.
deepspeed/​runtime/​fp16/​loss_scaler.py Allows scale backoff when hysteresis is zero or negative.

Review effort: Balanced

if self.delayed_shift == 1 or self.cur_hysteresis == 1:
# Back off once the hysteresis budget is used up. `<= 1` (not `== 1`) so that a configured
# hysteresis of 0 or below means "no delay" instead of never lowering the scale.
if self.delayed_shift <= 1 or self.cur_hysteresis <= 1:
@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 14, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fp16 hysteresis: 0 (or negative) makes DynamicLossScaler never lower the loss scale, so every overflowing step is skipped forever

2 participants