Skip to content

Fix DynamicLossScaler never lowering scale when hysteresis <= 0 - #8993

Closed
siddhant-rajhans wants to merge 1 commit into
deepspeedai:masterfrom
siddhant-rajhans:fix/dynamic-loss-scaler-hysteresis
Closed

siddhant-rajhans wants to merge 1 commit into
deepspeedai:masterfrom
siddhant-rajhans:fix/dynamic-loss-scaler-hysteresis

Conversation

@siddhant-rajhans

Copy link
Copy Markdown

When fp16.hysteresis is set to 0 or a negative value, DynamicLossScaler.update_scale tests self.cur_hysteresis == 1, which never triggers because the counter starts at 0 (or below) and only decrements further from 1. The loss scale is never reduced after overflow, causing every subsequent step to be skipped indefinitely.

Fix: change == 1 to <= 1 so that a hysteresis of 0 or below is treated as "no delay", matching the stage 0 fp16 optimizers (#8647). The default hysteresis: 2 is unaffected.

Testing: added tests/unit/runtime/test_dynamic_loss_scaler_hysteresis.py covering hysteresis values of 0, -1, -5, 1, and 2.

Fixes #8982

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T05:35:48.507246Z 35d643c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 35d643cee6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +37 to +40
def test_zero_or_negative_hysteresis_lowers_scale(self, delayed_shift):
scaler = _make_scaler(delayed_shift)
scaler.update_scale(overflow=True)
assert scaler.cur_scale < INIT_SCALE

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Add an engine-level regression for nonpositive hysteresis

For fp16.hysteresis <= 0, the user-visible fix is that ZeRO training can recover after overflow, but these tests invoke update_scale() directly and stub distributed rank. They never exercise configuration propagation, overflow detection, step skipping, or subsequent optimizer progress; the existing ZeRO overflow tests use positive hysteresis. Add a DistributedTest with SimpleModel through deepspeed.initialize() that verifies an overflowing step lowers the scale without updating weights, then a finite step updates weights for zero and negative hysteresis, and execute it on an fp16-capable device. Root AGENTS.md explicitly requires integration tests for training-loop contract changes and actual-device execution.

AGENTS.md reference: AGENTS.md:L81-L82

Useful? React with 👍 / 👎.

When `fp16.hysteresis` is set to 0 or a negative value, `update_scale`
tests `self.cur_hysteresis == 1` which never triggers because the
counter starts at 0 (or below) and only moves further away from 1.
This means the loss scale is never reduced after overflow, causing
every subsequent step to be skipped indefinitely.

Change the equality checks to `<= 1` so that a hysteresis of 0 or
below is treated as "no delay", matching the behavior of the stage 0
fp16 optimizers. The default `hysteresis: 2` is unaffected.

Fixes deepspeedai#8982

Signed-off-by: Siddhant Rajhans <51508945+siddhant-rajhans@users.noreply.github.com>
@siddhant-rajhans
siddhant-rajhans force-pushed the fix/dynamic-loss-scaler-hysteresis branch from 35d643c to a17d817 Compare November 4, 2026 08:52
@siddhant-rajhans

Copy link
Copy Markdown
Author

Closing this as a duplicate of #8983, which was opened first.

@gpu-test-labeler gpu-test-labeler Bot added the skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU. label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-gpu-test Requires positive evidence that every changed behavior can be validated without a GPU.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fp16 hysteresis: 0 (or negative) makes DynamicLossScaler never lower the loss scale, so every overflowing step is skipped forever

1 participant