Lag-aware failback checks for geo-redundant failover by mgravell · Pull Request #3242 · StackExchange/StackExchange.Redis

mgravell · 2026-09-17T13:57:12Z

Note

Implementation is in. Not gated behind an experiment: like the rest of
Availability/, it ships as regular API. Everything new is additive; the existing API is not changed apart
from three bug fixes (below).

What this adds

A group can now ask, before moving traffic to a member, whether that member is fit to take it, not only
whether it is alive. For Redis Enterprise Active-Active deployments, a built-in probe asks the cluster's own
database availability API whether the
member is available and caught up, so a group does not fail back onto a region that is reachable but stale.

member.FailbackHealthCheck = HealthCheck.LagAware(new LagAwareOptions
{
    Credentials = _ => new(new NetworkCredential(user, password)), // the cluster's management credentials
});

Core (group selection)

ConnectionGroupMember.FailbackHealthCheck and MultiGroupOptions.FailbackHealthCheck: consulted only for
members that are not active; only Healthy makes them eligible (Inconclusive does not).
Never evicts the active member. Liveness stays with HealthCheck.
Liveness fallback: if no live member is eligible (in a partition every member is behind at once), the
group still serves from the best live member. That member is provisional: it is left for an eligible
member as soon as one is available, and becomes an ordinary active member once it passes its own check.
Other clients never select a member that fails its lag check; this is the nearest equivalent that still
serves.
HealthCheckProbe.Scope / HealthCheckProbeScope: a probe can run once per member instead of once per
endpoint.

The probe (`HealthCheck.LagAware`, `HealthCheckProbe.LagAware`, `LagAwareOptions`, `LagCheckMode`)

Only the management credentials are required; everything else is derived or left to the cluster:

	default	why
REST endpoint	`https://{cluster}:9443/`, from a database host `redis-{port}.{cluster}`	only the cluster's master answers every route; other nodes redirect to its internal address
database id	discovered by host and port, filtered requests only	unfiltered `/v1/bdbs` returns database passwords, even to `db_viewer`
lag question	asked only of Active-Active databases	asked of any other database, the server reports it unavailable regardless of tolerance
tolerance	none sent; the cluster's own setting applies	the server says 100 ms, every other client 5000 ms
"could not ask"	`Inconclusive`, never `Unhealthy`	a management-plane problem must not fail anything over
certificate	`TrustIssuer(...)` / `CertificateValidation`	the 9443 certificate is self-signed by the cluster
client certificate	`ClientCertificate`, `SetUserPfxCertificate`, `SetUserPemCertificate`	mTLS to the REST API, as every other client offers
redirects	never followed; reported	following one hangs on an unreachable internal address

Credentials are re-read per request (rotation). Custom certificate validation and client certificates are
unavailable on net461. System.Text.Json is a new package reference on net461, net472 and
netstandard2.0 only (NRedisStack already depends on it).

Bug fixes in GA code

Each a separate commit with tests.

FailbackDelay = 0 on .NET Framework. ConnectionGroupMember.UpdateState cleared a failure only if it
was strictly before the cutoff (now - FailbackDelay). On .NET Framework DateTime.UtcNow advances about
every 15.6 ms, so a failure and the passing check that should clear it often share a tick, and the member was
held out for another health-check interval despite "immediate failback". Now inclusive; no measurable change
for any positive delay. Found because the new group tests failed on Windows net481 CI only.

HealthCheckProbePolicy: a run of nothing but Inconclusive probes ("could not tell") now reports
Inconclusive under every policy:

AllSuccess (the default) reported it as Healthy, contrary to its own comment. Harmless for liveness
(both count as connected), but it would have made "could not ask the REST API" read as "caught up".
AnySuccess reported it as Unhealthy, contrary to its own comment ("fail only when all probes have
failed"). That one did affect liveness: a probe that cannot tell (StringSet on a replica endpoint, for
example) marked its whole member unhealthy.

Impact on people who do not opt in (for the release notes)

The feature itself is inert unless configured. FailbackHealthCheck is null by default (member and group),
in which case every member counts as eligible, the provisional state is never entered, and selection is
exactly as before; the per-pass check loop finds nothing and returns before allocating. The existing group
tests pass unchanged. Nothing enables it automatically (no options provider changed).

What does change for everyone:

change	who sees it	effect
`System.Text.Json` package dependency	`net461`, `net472`, `netstandard2.0` consumers (in-box on `net8.0`+); `net461`/`net472` also gain a framework reference to `System.Net.Http`	a new dependency whether or not the feature is used: the usual .NET Framework version and binding-redirect exposure. Same 10.0.5 family as the existing dependencies there; NRedisStack already depends on it
`AnySuccess`: all-`Inconclusive` is `Inconclusive`, not `Unhealthy`	groups whose health check uses `AnySuccess`	can change selection: a member wrongly marked unhealthy (e.g. a key-write probe on a replica endpoint) may now be selected
`AllSuccess` (default): all-`Inconclusive` is `Inconclusive`, not `Healthy`	only code calling `HealthCheck.CheckHealthAsync` directly and testing for `Healthy`	none for group selection (`UpdateState` and endpoint collation treat both alike)
`FailbackDelay = 0` fails back within the same clock tick	.NET Framework group users, by default (zero is the default)	a member can return one health-check interval sooner after a failure; .NET Core already behaved this way

None of these is a reason not to merge, but the first, second and fourth belong in the release notes. The
three fixes could move to a separate PR if this one should change nothing for non-users; the dependency
cannot, since the probe lives in the main package.

How it was validated

notes/lag-aware/findings.md section 10 has the measurements; highlights, all on RS 8.0.22:

Pausing sync into one Active-Active member flips only that member's lag check, within seconds, while
its plain check stays 200; the tolerance is honoured on a running link (100 ms fails, 600000 ms passes
under a write burst).
Under a partition between the clusters, the lag check reports "caught up" for about a minute, then
fails on both sides. That is why the liveness fallback exists, and why the docs recommend a FailbackDelay
of at least that long.
The real probe, against live clusters: healthy on all four members; only the paused member unhealthy while
paused; Inconclusive (with a precise log line) for wrong credentials and for an untrusted certificate.
This run also found the master-redirect behaviour, which the stubbed tests could not.

Tests:

FailbackHealthCheckTests (selection rules) and LagAwareProbeTests (the probe against a stubbed transport
replaying captured response shapes); the availability suite passes on net10.0 and net8.0, and the new
tests on net481 under Mono.
LagAwareFailbackScenarioTests in the fault-injector tier: on a live Active-Active pair, the group held off
the preferred member for 15 s while it was behind (alive and connected), and failed back about 4 s after
sync resumed, by which time it had all of the writes. No network fault is used: sync is paused through the
cluster's REST API and resumed in a finally. Passed on three separate environments, the last time on this
branch as rebased.

Docs

docs/Failover.md gains "Failback health checks" and "Redis Enterprise: lag-aware availability", and its
AllSuccess / AnySuccess descriptions now say how an all-Inconclusive run is reported.

Open

Not verified against a server that requires it: an actual mutual-TLS handshake on the REST API (the test
clusters do not require client certificates on 9443). Loading and wiring are tested.
Worth raising with the Redis Enterprise team, from the measurements: read-only roles can read database
passwords from unfiltered reads; a lag failure is indistinguishable by body from "not Active-Active"; the
lag check is blind for about a minute when a link breaks; and every other client's defaults (lag check on,
5000 ms, no Active-Active check) would report every non-Active-Active database as permanently unhealthy
(inferred from their code).
A network_latency bug in the fault injector (its cleanup thread loses its SSH session, leaving the delay
in place) was reported to redis-developer/cae-client-testing.

Overlap to be aware of

notes/readme.md also appears in #3241,
which introduced the notes/ convention. Whichever merges second gets a one-line conflict in its Topics
list.

Confirms the premise: this is the client-side geographic failover feature family, and Lettuce, Jedis and redis-py have all shipped it. It feeds exactly the health checks our Availability/ namespace already has behind SER007. The abstractions line up almost one-to-one with redis-py's - probe, policy (all/any/majority), circuit breaker, per-member config - and two details make it land cleanly: HealthCheckProbe has no internal abstract members, so it is already subclassable from outside the assembly, and ConnectionGroupMember already carries a per-member HealthCheck override, which is what a per-cluster REST endpoint needs. Where we are better placed: HealthCheckResult has Inconclusive, which Lettuce's two-state HealthStatus lacks. Lettuce maps a failed REST call to UNHEALTHY, so a management-plane outage is indistinguishable from the data plane being down and can fail over a database that is serving fine. We can say "could not determine" instead. The awkward part is that src/ has no HTTP client and no JSON parser. JSON turns out to be avoidable if the bdb uid is configured rather than discovered, since the availability call is a status-code check. Also records the defaults disagreeing across clients: 100ms everywhere - server, docs, redis-py - but 5000ms in Lettuce.

mgravell offered to add one; the shape is what decides whether it helps. A member returning HttpMessageHandler would drag System.Net.Http into StackExchange.Redis's public API for the sake of one commercial deployment target - which is the thing keeping this in a separate package was meant to avoid, decided the expensive way. A stream-shaped member instead composes into SocketsHttpHandler.ConnectCallback, which takes exactly a ValueTask<Stream>, so the Enterprise side gets a complete HTTP hijack and core gains no HTTP types at all. Tunnel already speaks that language via BeforeAuthenticateAsync and ConnectTransportAsync, and adding a virtual is binary-safe for subclasses. Two functional arguments for Tunnel over a separate knob, beyond testing: a CONNECT proxy configured for Redis almost certainly applies to the Enterprise REST API too, so co-locating makes that correct by construction; and the test harness already assigns Tunnel, so one assignment would spoof both planes. Caveats recorded: ConnectCallback is net5+, the existing members are EndPoint/ConnectionType-shaped where HTTP wants a Uri, and widening Tunnel's remit means HttpProxyTunnel and LoggingTunnel each need an opinion about whether they apply to the management plane. Also retires open question 5: RawConfig and ConfigurationOptions.Tunnel are both already public, so a probe reaches the tunnel from the context it already gets - no HealthCheckContext change needed.

I had written that inducing "reachable but stale" on demand is not something anyone can do in CI, so the fakes were the only route to the scenario that matters. That is wrong. Redis ships a fault injection service that the client suites drive over HTTP - POST /action with a type and parameters, poll the returned action_id - and Lettuce's ActiveActiveFailoverScenarioTest injects network_latency against a bdb_id for a bounded duration, which is exactly the mechanism for pushing replication lag past the tolerance and watching the lag-aware check flip while the plain check stays green. So the authoritative end-to-end test is real cluster plus fault injector, on a road Lettuce has already built. The fakes are demoted rather than removed: they still cover machines with no Enterprise cluster, they are far faster than an injector action that budgets minutes plus a stabilisation wait, and they cover what the injector does not obviously reach - specific error_code bodies, 401/403 credential expiry, slow-but-successful responses for probe timeouts. Also records what the real clusters are uniquely good for: everything in section 1 is read off documentation, and a short session settles whether extend_check=lag is actually supported, what failure bodies really contain, whether the Host header is required, what the bdbs listing really returns - which decides whether the JSON-avoidance argument holds - and what lag real geo links show, which is the empirical way to settle 100ms versus 5000ms.

… questions It is not merged - tests/StackExchange.Redis.FaultInjector.Tests lives on marc/maint-optin-server, open as #3191 (107 files, +13160) - which is presumably why I did not find it when I grepped a branch taken fresh from main. Two pieces of it bear directly on this. FaultInjectorClient wraps the same POST /action + GET /action/{id} contract go-redis, redis-py and node-redis converged on, and adds GetValidTriggersAsync, so we can ask a deployment which lag-inducing effects it supports instead of guessing at an action name. And ClusterRestClient is already a Redis Enterprise REST client on 9443 - Basic auth, System.Text.Json, GET /v1/bdbs?fields=uid,name - so the availability call is a method on an existing class, not a new project. It also settles by observation what I had listed as open and sourced only from documentation: the management certificate is self-signed per environment and pinned to a CA from the config directory, so the probe needs CA configuration and cannot rely on the ambient trust store; auth really is Basic; and the bdbs listing returns the requested fields, so discovery is easy - which weakens the avoid-JSON argument to "it keeps JSON out of src/", not "discovery is hard". Sequencing consequence recorded: an end-to-end test is downstream of #3191. And #3191's own summary is the best argument for validating before building - same vendor, same class of API: the specifications are prose, the payloads were never published, and nearly every starting assumption was wrong.

Step 0 is to wait for #3191 to merge and build from there, rather than writing a second Redis Enterprise REST client against the same cluster and reconciling the two later - its ClusterRestClient already does Basic auth, CA pinning and GET /v1/bdbs, and its FaultInjectorClient can be asked which lag-inducing effects a deployment supports. The one thing that can start earlier is the API validation, because it produces notes rather than code. Step 1 is that validation, before any implementation: a checklist of things currently taken from documentation, including the measurement that settles 100ms versus 5000ms, with captured bodies saved as fixtures so the stubbed tests replay real payloads. Step 2 deliberately does not decide where the code lives, but records that the dependency argument is weaker than it looked - System.Net.Http needs no package reference on any target we ship - so the question is judgement, not constraint. Also records that this branch becomes the feature branch when step 0 clears, and that the way to take the merge is to rebase onto main rather than merge main in, since #3191 squash-merges.

…cess #3191 merged as dc915bb, so this branch is rebased onto it - which is what step 0 asked for, and rebased rather than merged because #3191 squash-merged and a merge would have moved the merge base without bringing its ancestry. The fault-injector tier is present here now. Also recorded where the work can happen. Step 1 is validation against a real deployment plus the fault injector, and both live on the machine holding the environment directory; this box has no ~/aws and no env_output.json anywhere, so nothing in step 1 could be run or verified here. Worth writing down rather than rediscovering, since the environment being absent looks identical to the environment being offline. Steps 2 onward are ordinary code and are not tied to a machine. Noted too that the directory is the AWS multi-cluster template, so env_output.json nests its outputs under .clusters.value[0] - a shape FaultInjectorEnvironment already handles, so it needs no work, just no surprise.

… config CA The 9443 certificate is self-signed by the cluster itself; the CA in the config directory only signs the RESP proxy certificates, so the chain build could never pass. Pin the leaf on first use, keyed by host so multi-cluster templates work.

…t says there Records the real-deployment results (findings section 10): off Active-Active the lag check fails unconditionally; on it, the stale member flips within seconds of sync pausing and the tolerance is honoured on a running link. Also the two integration problems found in Availability/, which mean src/ needs a change.

…to 5000 ms Read from source: redis-py's 5000 contradicts its own docstring's 100. None of the three checks for Active-Active, and none has a third result state.

…ewer suffices Active-Active membership has no RESP signal, so it comes from REST (fields=crdt) or configuration. db_viewer can call everything a probe needs, but unfiltered bdb/crdb reads expose database passwords even to it, so only filtered requests. The REST endpoint derives from the configured database hostname.

…base host TrustIssuer(leaf) via TlsOptions validates the 9443 call unchanged. No other client activates lag checks automatically or checks crdt; go-redis routes the local form through each database host, which makes it truthful, and scopes the lag verdict to failback only.

Same verdict, same timing, stale member only, tolerance ignored while paused. Its failure code is bdb_endpoint_unavailable, shared with the wrong-node case, so the choice between forms is about call shape, not information.

A failback-only health check per member rather than a new result value, a member-scoped probe, and a LagAware factory whose only required input is the management credential. Compared against Lettuce, Jedis, redis-py and go-redis.

…her clients Keep LagAware naming and failback (not fallback) terminology; rotating NetworkCredential callback plus a separate client-certificate option; and keep the liveness fallback, which no client has, because a dead peer probably makes the survivor's lag check fail.

…th sides Partitioning the clusters (tagged iptables rules, self-healing) showed no lag for ~60 s while sync had stopped, then tolerance-governed failure on both members; a tolerance of 0 is honoured. Supports the liveness fallback, and adds a caveat: failback in the first minute after a link breaks is not protected.

The default policy reported Healthy when no probe failed, even if none succeeded. Harmless for liveness, where both count as connected, but wrong for any caller that distinguishes them.

…ing to it (SER011) ConnectionGroupMember.FailbackHealthCheck (and a group default) is consulted only for members that are not active, and only Healthy makes them eligible. It never evicts the active member, and when no live member is eligible, selection falls back to liveness, since a partition leaves every region behind at once. HealthCheckProbe.Scope lets a probe run once per member rather than per endpoint.

…ble and caught up (SER011) A member-scoped probe of the database availability API. Only the management credentials are required: the REST endpoint derives from the configured host, the database is discovered with filtered requests only, the lag question is asked only of Active-Active databases, and no tolerance is sent unless set. Could-not-ask is Inconclusive, never Unhealthy. System.Text.Json on the down-level targets; custom certificate validation is unavailable on net461.

Found against live clusters: non-master nodes answer the management routes (discovery, the crdt lookup) with a 307 to the master's internal address, so deriving the REST endpoint from the database host timed out for three of four members. Derive the cluster FQDN (strip redis-{port}.), which resolves to the master, and report a redirect as Inconclusive rather than following it.

…ve member Pauses sync into the preferred member through its cluster's REST API while writes go to the other, and asserts the group stays off it for 15s although it is alive, then fails back once it has caught up. The tier now reads every cluster's credentials, and shares its certificate pin with the probe.

LagAwareOptions.ClientCertificate, with SetUserPfxCertificate and (net6.0+) SetUserPemCertificate mirroring ConfigurationOptions and sharing its loaders. Wired into SocketsHttpHandler on .NET and HttpClientHandler down-level; not supported on net461. Every other client offers mTLS for this call.

Other clients never select a member that fails its lag check; the nearest equivalent that still serves when every member is behind is to leave such a member for an eligible one as soon as one is available, whatever the weights, and to keep checking it so it becomes an ordinary active member once caught up.

main now uses 124 and 125 (#3257); the guard test from #3266 rejects duplicates.

AnySuccess reported Unhealthy whenever nothing succeeded, even when nothing failed either - so a probe that cannot tell (a key-write probe on a replica, say) marked its member unhealthy. Now Inconclusive, matching AllSuccess and MajoritySuccess.

main has since taken 128 as well.

…k; no ValueTuple UpdateState compared the last failure strictly before the cutoff. With FailbackDelay zero the cutoff is now, and on .NET Framework's ~15.6ms clock a failure and the passing check that clears it often share a tick, holding the member out for another interval. Compare inclusively. Also replace a named tuple in the failback-check pass: the library must not reference System.ValueTuple.

Remove SER011: the attributes, the constant and NoWarn entry, the [SER011] API prefixes and docs/exp/SER011.md. It never shipped, so it is not a reserved id. Also restores main's using directives and encoding in the files that already existed there, which earlier edits had disturbed.

mgravell force-pushed the marc/lag-aware-availability branch from cce049c to 916a563 Compare September 18, 2026 04:34

mgravell changed the title ~~[blocked on #3191] Lag-aware availability checks for geo-redundant failover: findings and plan~~ Lag-aware availability checks for geo-redundant failover: findings and plan Oct 6, 2026

mgravell force-pushed the marc/lag-aware-availability branch from 701beba to 10b2c0b Compare October 6, 2026 16:03

mgravell changed the title ~~Lag-aware availability checks for geo-redundant failover: findings and plan~~ Lag-aware failback checks for geo-redundant failover (SER011) Oct 9, 2026

mgravell force-pushed the marc/lag-aware-availability branch from 74c4b82 to 010f62c Compare October 13, 2026 10:24

mgravell added 25 commits October 13, 2026 11:25

Correct the client defaults: Lettuce, Jedis and redis-py all default …

0420722

…to 5000 ms Read from source: redis-py's 5000 contradicts its own docstring's 100. None of the three checks for Active-Active, and none has a third result state.

Sketch the lag-aware API for review

bfb7844

A failback-only health check per member rather than a new result value, a member-scoped probe, and a LagAware factory whose only required input is the management credential. Compared against Lettuce, Jedis, redis-py and go-redis.

AllSuccess: a run of only inconclusive probes is not a pass

796ca5e

The default policy reported Healthy when no probe failed, even if none succeeded. Harmless for liveness, where both count as connected, but wrong for any caller that distinguishes them.

Notes: core change implemented behind SER011 (SER007 is retired)

2ba0a30

Notes: probe implemented; RawConfig is not public after all

2c647d7

Notes: management routes redirect to the master's internal address

3950797

Notes: tier-3 scenario test passing

8e135d1

mgravell added 5 commits October 13, 2026 11:25

Notes: fallback choice is provisional, as decided by client parity

fa5b8ab

Move the lag-aware log message to a free event id

4fce481

main now uses 124 and 125 (#3257); the guard test from #3266 rejects duplicates.

Move the lag-aware log message to a free event id (again)

059f7dc

main has since taken 128 as well.

mgravell force-pushed the marc/lag-aware-availability branch from 010f62c to 059f7dc Compare October 13, 2026 10:29

mgravell marked this pull request as ready for review October 13, 2026 13:08

mgravell added 2 commits October 13, 2026 14:16

Notes: no experiment

2c0540b

mgravell changed the title ~~Lag-aware failback checks for geo-redundant failover (SER011)~~ Lag-aware failback checks for geo-redundant failover Oct 13, 2026

Notes: drop the SER011 history

2c8c9c3

mgravell merged commit d59ff9e into main Oct 13, 2026
5 checks passed

mgravell deleted the marc/lag-aware-availability branch October 13, 2026 13:44

Navigation Menu

Repository navigation

Lag-aware failback checks for geo-redundant failover - #3242

Lag-aware failback checks for geo-redundant failover#3242
mgravell merged 34 commits into
mainfrom
marc/lag-aware-availability

mgravell commented Sep 17, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mgravell commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this adds

Core (group selection)

The probe (HealthCheck.LagAware, HealthCheckProbe.LagAware, LagAwareOptions, LagCheckMode)

Bug fixes in GA code

Impact on people who do not opt in (for the release notes)

How it was validated

Docs

Open

Overlap to be aware of

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mgravell commented Sep 17, 2026 •

edited

Loading

The probe (`HealthCheck.LagAware`, `HealthCheckProbe.LagAware`, `LagAwareOptions`, `LagCheckMode`)