Skip to content

fix: Use monotonic clocks for stream-init durations and FDv2 state windows - #537

Merged
tanderson-ld merged 1 commit into
mainfrom
tanderson/sdk-3105-monotonic-clock
Oct 12, 2026
Merged

tanderson-ld merged 1 commit into
mainfrom
tanderson/sdk-3105-monotonic-clock

Conversation

@tanderson-ld

@tanderson-ld tanderson-ld commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replaces wall-clock time with monotonic time in the two remaining places this SDK measures a duration or decides behavior from one: stream-init diagnostics and the FDv2 fallback/recovery windows. Wall clocks remain only for values reported outside the process or compared against externally supplied times. Implements SDK-3105.

A wall-clock step (NTP correction, manual clock set, container/VM time resync, the time correction when a frozen environment thaws) previously could cause:

  • Stream-init diagnostics (datasource/streaming.py, datasource/async_streaming.py): a forward step inflated the reported durationMillis by the step size; a backward step hit the elapsed >= 0 else 0 clamp and silently reported zero — a lost measurement either way.
  • FDv2 fallback/recovery verdicts (datasystem/fdv2_common.py, 60s/10s/300s windows): a forward step could make a just-interrupted source look an hour degraded and trigger an immediate synchronizer fallback; a backward step could suppress a legitimate one.

Two audit findings in this story's original scope — the wall-clock schedulers in repeating_task.py and aio/concurrency.py — were already resolved on main by the RepeatingTask/DelaySource rework that shipped in 9.18.0 (the wait is event-based with no clock arithmetic), so they appear here as this note rather than as code.

Changes

  • Impl::util.monotonic_seconds() (time.monotonic()) is the single monotonic seam.
  • Stream-init: the connection-attempt stamp is now monotonic and serves as both the duration base and the "attempt in flight" sentinel (explicit is not None, which also fixes a latent float-truthiness wart in the old guard). The reported timestamp stays wall-clock; the negative-value clamp is removed because a negative duration is now impossible.
  • FDv2 windows: the condition-check loop in each coordinator owns a small _StateAgeTracker — created per synchronizer session, so windows reset naturally on failover — which measures time-in-state monotonically from first observation. Its identity key is (state, since): a real transition changes since even when the state repeats (an interrupted→valid→interrupted flap between two 10-second checks resets the window), while a same-state error update preserves it (a stream of error reports cannot keep resetting the clock). since is used only for equality, never arithmetic. fallback_condition/recovery_condition take the measured seconds as a parameter and are pure, table-testable predicates. This observer-owned shape matches the Rust server SDK's FDv2 implementation, which builds its failover deadlines from Instant::now() inside the coordinator.
  • The status provider and the public DataSourceStatus are untouched — zero diff against main.

Deliberately unchanged (wall clock is correct): DataSourceStatus.since, DataSourceErrorInfo.time, event creationDates, diagnostic payload timestamps, debugEventsUntilDate comparisons, and big-segments staleness (cross-process stamp).

Behavioral notes

Not a breaking change — every modified component is Impl-internal; the public status surface is untouched; behavior under a stable clock is identical, with one bounded exception:

  1. Observation latency. Time-in-state is measured from when the check loop first sees a state, not from the transition itself, so a verdict can fire up to one 10-second check period later than before (material only for the 10-second INITIALIZING window, where worst-case detection moves from ~20s to ~30s). Always in the safe direction — defer, never spuriously trigger.
  2. Suspend semantics. time.monotonic() on Linux does not advance during system suspend, so these measurements now count process-visible time. Notably this costs nothing in AWS Lambda: a frozen execution environment pauses the entire microVM with no guest-visible suspend, so every in-process clock behaves identically there — while the thaw-time wall correction is exactly the kind of step this change defends against.
  3. Test suites that freeze time. Applications using freezegun (or similar) to drive SDK-internal timing should note these measurements no longer follow time.time(); freezegun does not patch time.monotonic() by default.

Known sibling, deliberately out of scope

ldclient/impl/datasourcev2/streaming.py and async_streaming.py carry the same wall-clock stream-init pattern (including a wall-clock future-start stamp from classify_stream_error). Those files postdate the originating audit; they'll be handled under a separate decision rather than folded in here.

Relationship to the Ruby change

Same change-class as launchdarkly/ruby-server-sdk#459. The FDv2 window fix intentionally uses a different shape than Ruby's (observer-owned tracker here vs. a provider-held stamp there); the Rust FDv2 implementation is the shared precedent for the approach used here.

Testing

Full suite passes (1766 passed, 371 skipped — database integrations). New test_fdv2_conditions.py covers the threshold truth table (strict-> boundaries pinned), the tracker's first-observation/flap/error-spam semantics, and a regression test proving a +1h wall step can no longer trigger a spurious fallback. Wall-step regression tests added to both streaming test suites use a forward step deliberately: the old code clamped negatives to zero, so only the forward direction distinguishes old from new behavior. mypy, isort, and pycodestyle clean.


Note

Overview
Replaces wall-clock timing with monotonic measurements for two internal SDK behaviors so NTP or manual clock jumps cannot skew diagnostics or failover decisions.

Stream-init diagnostics (sync and async streaming processors): connection-attempt duration is computed from monotonic_seconds() instead of time.time(), while reported timestamps stay wall-clock. The old negative-duration clamp is removed; retry backoff is excluded from measured latency by stamping the attempt start after the wait.

FDv2 fallback/recovery (fdv2_common, wired in sync/async coordinators): a per-session _StateAgeTracker measures how long the current data-source state has been observed on the monotonic timeline (keyed by (state, since) so error spam does not reset the window). fallback_condition / recovery_condition now take that measured seconds_in_state instead of time.time() - status.since.

Adds monotonic_seconds() in impl/util.py and regression tests for wall-clock steps and FDv2 thresholds.

Reviewed by Cursor Bugbot for commit ee290ec. Bugbot is set up for automated code reviews on this repo. Configure here.

…ndows

Wall-clock steps (NTP corrections, manual clock sets, VM/container time
resync, thaw-time corrections in frozen environments) could inflate the
stream-init durations reported in diagnostic events (or silently drop
them via the negative-value clamp), and could trigger or suppress the
FDv2 fallback/recovery verdicts by distorting the measured time in
state.

Stream-init durations are now measured on the monotonic clock, with the
wall clock kept only for the reported timestamp. The FDv2 coordinators
measure time in state with a small observer (_StateAgeTracker) owned by
the condition-check loop, keyed on (state, since) so flaps reset the
window and same-state error reports do not.

Note for applications that use freezegun or similar to drive
SDK-internal timing: these measurements no longer follow time.time().

SDK-3105
@tanderson-ld
tanderson-ld requested a review from a team as a code owner October 9, 2026 14:05
Comment thread ldclient/impl/util.py
@tanderson-ld
tanderson-ld requested a review from joker23 October 12, 2026 15:08
@tanderson-ld
tanderson-ld merged commit e8501c5 into main Oct 12, 2026
18 checks passed
@tanderson-ld
tanderson-ld deleted the tanderson/sdk-3105-monotonic-clock branch October 12, 2026 18:07
tanderson-ld pushed a commit that referenced this pull request Oct 13, 2026
🤖 I have created a release *beep* *boop*
---


##
[9.18.2](9.18.1...9.18.2)
(2026-10-12)


### Bug Fixes

* Use monotonic clocks for stream-init durations and FDv2 state windows
([#537](#537))
([e8501c5](e8501c5))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> **Release 9.18.2** bumps the published SDK version from **9.18.1** to
**9.18.2** across the manifest (`VERSION`, `pyproject.toml`, Release
Please manifest, and `PROVENANCE.md` verification example).
> 
> The changelog documents the shipped fix
([#537](#537)):
**stream initialization duration** and **FDv2 state window** timing now
use **monotonic clocks** instead of wall-clock time, so NTP adjustments
or clock skew are less likely to skew timeouts and internal timing.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
f8945a5. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
tanderson-ld added a commit that referenced this pull request Oct 26, 2026
…timing (#539)

## Summary

Switches the two remaining wall-clock duration measurements in this SDK
to the monotonic clock, found by the clock-monotonicity re-audit.
Implements SDK-3307 (re-audit findings #39/#40/#41).

This is a **pure clock-source substitution with no observable behavioral
change versus `main`.** The recording structure, control flow, and
public contracts are byte-for-byte `main`; only the clock a duration is
measured from changes. A wall-clock step (DST via naive
`datetime.now()`, NTP correction, VM/container time resync) can no
longer distort these durations.

## Changes

- `Impl::util.monotonic_seconds()` is the single monotonic seam (added
in the sibling clock-monotonicity work; reused here).
- **Migration op latency** (`migrations/migrator.py`,
`async_migrator.py`): measured with `monotonic_seconds()` and delivered
to `OpTracker.latency` as a `timedelta` — identical type and, under a
stable clock, identical value; a backward step can no longer produce a
negative latency. Finding #39 (HIGH).
- **FDv2 stream-init diagnostics** (`datasourcev2/streaming.py`,
`async_streaming.py`, `streaming_common.py`): the connection-attempt
baseline and the `next_attempt_start` projection now use
`monotonic_seconds()`, and the duration is computed on the monotonic
clock as a difference of floored milliseconds (matching `main`'s
rounding exactly). The wall clock is retained for the reported
`timestamp` (via `current_time_millis()`) and for the three
`DataSourceErrorInfo.time` error timestamps. Findings #40 / #41 (MED).
- The recording lifecycle is unchanged from `main`: the projected
`next_attempt_start` baseline, the per-site `elapsed if elapsed >= 0
else 0` clamp, the record-iff-baseline-set guard, and all
`Start`/`Fault` control flow are preserved. `diagnostics.py` is
unchanged.

**Deliberately unchanged** (wall clock is correct — values communicated
outside the process): `DataSourceErrorInfo.time`, event `creationDate`,
diagnostic `dataSinceDate`, the big-segments cross-process staleness
stamp.

## Explicitly out of scope

The pre-existing FDv2 stream-init projection inaccuracy (a reconnect's
recorded duration can be driven to the clamp by the backoff projection)
and the related diagnostic-count behavior during a sustained outage
exist identically on `main` and are **not** changed by this PR. An
earlier revision of this PR reworked that behavior; it has been reverted
so this change is a clock swap only. Those are diagnostics-only, affect
no flag evaluation, data correctness, or retry behavior, and will be
addressed separately if desired.

## Behavioral notes

Not a breaking change — every modified component is `Impl`-internal, and
behavior under a stable clock is unchanged. Note for test suites that
time-travel: these measurements no longer follow `time.time()`;
`freezegun` does not patch `time.monotonic()` by default.

## Testing

Full suite passes (1774 passed, 371 DB-skipped); mypy, isort,
pycodestyle clean. New tests assert the clock property without asserting
new behavior: migration latency measured monotonically (sync + async,
via a module-patched clock); FDv2 stream-init duration measured on the
monotonic clock while the reported timestamp follows the wall clock
(proven by advancing the monotonic clock while stepping the wall clock
backward), plus the preserved negative-elapsed clamp and
baseline-cleared skip. The migration latency assertions use a 50 ms
floor to tolerate coarse monotonic granularity on pre-3.13 Windows.

A three-agent review (general, security, adversarial) plus proof tests
confirmed no observable behavioral change versus `main`.

Same change-class as the merged python PR #537.

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Replaces wall-clock duration measurements with **monotonic** timing in
two internal areas so NTP/DST jumps no longer skew reported durations.
> 
> **Migration operations** (`migrator.py`, `async_migrator.py`): op
latency is measured with `monotonic_seconds()` and passed to the tracker
as a `timedelta` instead of `datetime.now()` deltas.
> 
> **FDv2 streaming** (`streaming.py`, `async_streaming.py`,
`streaming_common.py`): connection-attempt baselines, retry
`next_attempt_start` projections, and stream-init **elapsed** time use
the monotonic clock; diagnostic **timestamps** still use
`current_time_millis()`, and error `DataSourceErrorInfo.time` stays on
the wall clock. The existing negative-elapsed clamp and “record only
when baseline is set” behavior are unchanged.
> 
> Adds focused tests for monotonic stream-init recording (sync + async)
and migration executor latency; integration tests relax the minimum
latency assertion (100ms → 50ms) to account for coarse monotonic
resolution on some platforms.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
0b804ff. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants