Repository navigation
fix: Use monotonic clocks for stream-init durations and FDv2 state windows - #537
Merged
Merged
Conversation
…ndows Wall-clock steps (NTP corrections, manual clock sets, VM/container time resync, thaw-time corrections in frozen environments) could inflate the stream-init durations reported in diagnostic events (or silently drop them via the negative-value clamp), and could trigger or suppress the FDv2 fallback/recovery verdicts by distorting the measured time in state. Stream-init durations are now measured on the monotonic clock, with the wall clock kept only for the reported timestamp. The FDv2 coordinators measure time in state with a small observer (_StateAgeTracker) owned by the condition-check loop, keyed on (state, since) so flaps reset the window and same-state error reports do not. Note for applications that use freezegun or similar to drive SDK-internal timing: these measurements no longer follow time.time(). SDK-3105
joker23
reviewed
Oct 9, 2026
joker23
approved these changes
Oct 12, 2026
tanderson-ld
pushed a commit
that referenced
this pull request
Oct 13, 2026
🤖 I have created a release *beep* *boop* --- ## [9.18.2](9.18.1...9.18.2) (2026-10-12) ### Bug Fixes * Use monotonic clocks for stream-init durations and FDv2 state windows ([#537](#537)) ([e8501c5](e8501c5)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > **Release 9.18.2** bumps the published SDK version from **9.18.1** to **9.18.2** across the manifest (`VERSION`, `pyproject.toml`, Release Please manifest, and `PROVENANCE.md` verification example). > > The changelog documents the shipped fix ([#537](#537)): **stream initialization duration** and **FDv2 state window** timing now use **monotonic clocks** instead of wall-clock time, so NTP adjustments or clock skew are less likely to skew timeouts and internal timing. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit f8945a5. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
tanderson-ld
added a commit
that referenced
this pull request
Oct 26, 2026
…timing (#539) ## Summary Switches the two remaining wall-clock duration measurements in this SDK to the monotonic clock, found by the clock-monotonicity re-audit. Implements SDK-3307 (re-audit findings #39/#40/#41). This is a **pure clock-source substitution with no observable behavioral change versus `main`.** The recording structure, control flow, and public contracts are byte-for-byte `main`; only the clock a duration is measured from changes. A wall-clock step (DST via naive `datetime.now()`, NTP correction, VM/container time resync) can no longer distort these durations. ## Changes - `Impl::util.monotonic_seconds()` is the single monotonic seam (added in the sibling clock-monotonicity work; reused here). - **Migration op latency** (`migrations/migrator.py`, `async_migrator.py`): measured with `monotonic_seconds()` and delivered to `OpTracker.latency` as a `timedelta` — identical type and, under a stable clock, identical value; a backward step can no longer produce a negative latency. Finding #39 (HIGH). - **FDv2 stream-init diagnostics** (`datasourcev2/streaming.py`, `async_streaming.py`, `streaming_common.py`): the connection-attempt baseline and the `next_attempt_start` projection now use `monotonic_seconds()`, and the duration is computed on the monotonic clock as a difference of floored milliseconds (matching `main`'s rounding exactly). The wall clock is retained for the reported `timestamp` (via `current_time_millis()`) and for the three `DataSourceErrorInfo.time` error timestamps. Findings #40 / #41 (MED). - The recording lifecycle is unchanged from `main`: the projected `next_attempt_start` baseline, the per-site `elapsed if elapsed >= 0 else 0` clamp, the record-iff-baseline-set guard, and all `Start`/`Fault` control flow are preserved. `diagnostics.py` is unchanged. **Deliberately unchanged** (wall clock is correct — values communicated outside the process): `DataSourceErrorInfo.time`, event `creationDate`, diagnostic `dataSinceDate`, the big-segments cross-process staleness stamp. ## Explicitly out of scope The pre-existing FDv2 stream-init projection inaccuracy (a reconnect's recorded duration can be driven to the clamp by the backoff projection) and the related diagnostic-count behavior during a sustained outage exist identically on `main` and are **not** changed by this PR. An earlier revision of this PR reworked that behavior; it has been reverted so this change is a clock swap only. Those are diagnostics-only, affect no flag evaluation, data correctness, or retry behavior, and will be addressed separately if desired. ## Behavioral notes Not a breaking change — every modified component is `Impl`-internal, and behavior under a stable clock is unchanged. Note for test suites that time-travel: these measurements no longer follow `time.time()`; `freezegun` does not patch `time.monotonic()` by default. ## Testing Full suite passes (1774 passed, 371 DB-skipped); mypy, isort, pycodestyle clean. New tests assert the clock property without asserting new behavior: migration latency measured monotonically (sync + async, via a module-patched clock); FDv2 stream-init duration measured on the monotonic clock while the reported timestamp follows the wall clock (proven by advancing the monotonic clock while stepping the wall clock backward), plus the preserved negative-elapsed clamp and baseline-cleared skip. The migration latency assertions use a 50 ms floor to tolerate coarse monotonic granularity on pre-3.13 Windows. A three-agent review (general, security, adversarial) plus proof tests confirmed no observable behavioral change versus `main`. Same change-class as the merged python PR #537. <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Replaces wall-clock duration measurements with **monotonic** timing in two internal areas so NTP/DST jumps no longer skew reported durations. > > **Migration operations** (`migrator.py`, `async_migrator.py`): op latency is measured with `monotonic_seconds()` and passed to the tracker as a `timedelta` instead of `datetime.now()` deltas. > > **FDv2 streaming** (`streaming.py`, `async_streaming.py`, `streaming_common.py`): connection-attempt baselines, retry `next_attempt_start` projections, and stream-init **elapsed** time use the monotonic clock; diagnostic **timestamps** still use `current_time_millis()`, and error `DataSourceErrorInfo.time` stays on the wall clock. The existing negative-elapsed clamp and “record only when baseline is set” behavior are unchanged. > > Adds focused tests for monotonic stream-init recording (sync + async) and migration executor latency; integration tests relax the minimum latency assertion (100ms → 50ms) to account for coarse monotonic resolution on some platforms. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 0b804ff. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces wall-clock time with monotonic time in the two remaining places this SDK measures a duration or decides behavior from one: stream-init diagnostics and the FDv2 fallback/recovery windows. Wall clocks remain only for values reported outside the process or compared against externally supplied times. Implements SDK-3105.
A wall-clock step (NTP correction, manual clock set, container/VM time resync, the time correction when a frozen environment thaws) previously could cause:
datasource/streaming.py,datasource/async_streaming.py): a forward step inflated the reporteddurationMillisby the step size; a backward step hit theelapsed >= 0 else 0clamp and silently reported zero — a lost measurement either way.datasystem/fdv2_common.py, 60s/10s/300s windows): a forward step could make a just-interrupted source look an hour degraded and trigger an immediate synchronizer fallback; a backward step could suppress a legitimate one.Two audit findings in this story's original scope — the wall-clock schedulers in
repeating_task.pyandaio/concurrency.py— were already resolved on main by theRepeatingTask/DelaySourcerework that shipped in 9.18.0 (the wait is event-based with no clock arithmetic), so they appear here as this note rather than as code.Changes
Impl::util.monotonic_seconds()(time.monotonic()) is the single monotonic seam.is not None, which also fixes a latent float-truthiness wart in the old guard). The reportedtimestampstays wall-clock; the negative-value clamp is removed because a negative duration is now impossible._StateAgeTracker— created per synchronizer session, so windows reset naturally on failover — which measures time-in-state monotonically from first observation. Its identity key is(state, since): a real transition changessinceeven when the state repeats (an interrupted→valid→interrupted flap between two 10-second checks resets the window), while a same-state error update preserves it (a stream of error reports cannot keep resetting the clock).sinceis used only for equality, never arithmetic.fallback_condition/recovery_conditiontake the measured seconds as a parameter and are pure, table-testable predicates. This observer-owned shape matches the Rust server SDK's FDv2 implementation, which builds its failover deadlines fromInstant::now()inside the coordinator.DataSourceStatusare untouched — zero diff against main.Deliberately unchanged (wall clock is correct):
DataSourceStatus.since,DataSourceErrorInfo.time, eventcreationDates, diagnostic payload timestamps,debugEventsUntilDatecomparisons, and big-segments staleness (cross-process stamp).Behavioral notes
Not a breaking change — every modified component is
Impl-internal; the public status surface is untouched; behavior under a stable clock is identical, with one bounded exception:time.monotonic()on Linux does not advance during system suspend, so these measurements now count process-visible time. Notably this costs nothing in AWS Lambda: a frozen execution environment pauses the entire microVM with no guest-visible suspend, so every in-process clock behaves identically there — while the thaw-time wall correction is exactly the kind of step this change defends against.time.time(); freezegun does not patchtime.monotonic()by default.Known sibling, deliberately out of scope
ldclient/impl/datasourcev2/streaming.pyandasync_streaming.pycarry the same wall-clock stream-init pattern (including a wall-clock future-start stamp fromclassify_stream_error). Those files postdate the originating audit; they'll be handled under a separate decision rather than folded in here.Relationship to the Ruby change
Same change-class as launchdarkly/ruby-server-sdk#459. The FDv2 window fix intentionally uses a different shape than Ruby's (observer-owned tracker here vs. a provider-held stamp there); the Rust FDv2 implementation is the shared precedent for the approach used here.
Testing
Full suite passes (1766 passed, 371 skipped — database integrations). New
test_fdv2_conditions.pycovers the threshold truth table (strict->boundaries pinned), the tracker's first-observation/flap/error-spam semantics, and a regression test proving a +1h wall step can no longer trigger a spurious fallback. Wall-step regression tests added to both streaming test suites use a forward step deliberately: the old code clamped negatives to zero, so only the forward direction distinguishes old from new behavior. mypy, isort, and pycodestyle clean.Note
Overview
Replaces wall-clock timing with monotonic measurements for two internal SDK behaviors so NTP or manual clock jumps cannot skew diagnostics or failover decisions.
Stream-init diagnostics (sync and async streaming processors): connection-attempt duration is computed from
monotonic_seconds()instead oftime.time(), while reported timestamps stay wall-clock. The old negative-duration clamp is removed; retry backoff is excluded from measured latency by stamping the attempt start after the wait.FDv2 fallback/recovery (
fdv2_common, wired in sync/async coordinators): a per-session_StateAgeTrackermeasures how long the current data-source state has been observed on the monotonic timeline (keyed by(state, since)so error spam does not reset the window).fallback_condition/recovery_conditionnow take that measuredseconds_in_stateinstead oftime.time() - status.since.Adds
monotonic_seconds()inimpl/util.pyand regression tests for wall-clock steps and FDv2 thresholds.Reviewed by Cursor Bugbot for commit ee290ec. Bugbot is set up for automated code reviews on this repo. Configure here.