Repository navigation
fix(flags): give waiting callers one shared retry for inconclusive missing-flag requests - #259
marandaneto wants to merge 15 commits into
Conversation
posthog-elixir Compliance ReportDate: 2026-11-18T13:14:17.499773+00:00 ✅ All Tests Passed!47/47 tests passed Capture Tests✅ 30/30 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
|
Each key has at most one coordination round with a single shared retry. Callers keep the round id they joined, so they skip a key once its round ends instead of joining a later probe. Releases and give-ups are scoped to the coordinated call that made them.
…s to their request
afdb86b to
4487e94
Compare
dustinbyrne
left a comment
There was a problem hiding this comment.
I verified that a clean shared retry can avoid a second waiter’s request. Two regressions need correction before approval: bulk release does quadratic work in the coordinator, and a context-filtered retry can make an unfiltered caller miss an existing flag. Both reproduced through evaluate_flags/2 on this head and passed on base. Details and suggested fixes are inline.
AI-assisted review.
| end | ||
|
|
||
| defp maybe_forget(state, pid) do | ||
| leads? = Enum.any?(state.owners, fn {_key, owner} -> owner == pid end) |
There was a problem hiding this comment.
1. [P1] Bulk probe release does quadratic work on the coordinator
IMPORTANT · REQUIRED · Introduced — lib/posthog/feature_flags/missing_probe_coordinator.ex:387, RIGHT
When many FeatureFlags.evaluate_flags(name, %{distinct_id: id, flag_keys: keys}) calls overlap on a bulk scope missing from loaded local definitions, even a clean, prompt /flags response can leave the instance’s coordinator busy for hundreds of milliseconds. Releasing a probe should cost roughly the work of notifying its waiters, not scanning all active owners for each notification.
notify_waiters/3 calls maybe_forget/2 once per waiter per key. Head scans state.owners before checking whether the waiter still has an active wait or round membership. With K owned keys and W waiters who own none, that visits about W × K × (K−1) / 2 entries. Base checked the caller’s own owned set instead.
Executed regression (CASE=perf): Gate one clean omission until 50 waiters have registered, then drain the group. At 1,000 keys, base used 1.39M coordinator reductions / 44 ms; head used 227.25M / 496 ms. Doubling the scope from 500 to 1,000 keys roughly doubled base work and nearly quadrupled head work, so at_1000 < 3 * at_500 passed on base and failed on head. Both revisions made one remote request and returned the expected empty snapshots. A disjoint control also showed about 37K → 1.52M reductions for 50 one-key calls with 1,000 unrelated owners live; that control includes membership cleanup as well as ownership scans.
The 1,000-key scope is within the supported coordination limit. This is a stress case, not a production latency forecast; no deadline miss was measured. The coordinator processes the release serially, so unrelated calls can queue behind it. Consider checking wait and membership state first, restoring process-local owned-key tracking for cleanup, and skipping membership cleanup for claims that never joined a round. Keep the reductions test and the one-request clean-omission assertion.
| {remote_results, malformed?} = build_remote_results(scoped_remote_flags, response_body) | ||
| put_probe_returned(Map.keys(scoped_remote_flags)) | ||
|
|
||
| update_negative_knowledge( |
There was a problem hiding this comment.
2. [P1] A context-filtered retry can hide an existing flag from an unfiltered caller
IMPORTANT · REQUIRED · Amplifies a pre-existing omission classifier — lib/posthog/feature_flags.ex:630, RIGHT
Suppose loaded local definitions are stale and lack a flag tagged production. An initial probe fails, and these two callers are waiting on its key:
# In separate caller processes, with local evaluation enabled:
PostHog.FeatureFlags.evaluate_flags(name, %{
distinct_id: "staging-user",
flag_keys: ["checkout"],
evaluation_contexts: ["staging"]
})
PostHog.FeatureFlags.evaluate_flags(name, %{
distinct_id: "normal-user",
flag_keys: ["checkout"]
})If the staging caller leads the shared retry, its clean response legitimately omits checkout for that request. Head can record that omission as instance-wide absence. The normal caller then skips /flags and gets an empty snapshot instead of its enabled flag; later unfiltered callers may also miss it until refresh, eviction, or a positive response clears it. Base released the staging waiter without ownership after the first error, so its response could not write that absence. The initial-owner version of the classifier bug already existed; the retry creates a new path to it.
The official /flags API documents evaluation_contexts. At backend 036a0ced6d1ef86987ea28f692c8161c26586967, flags/flag_request.rs:100–101 accepts it and the evaluation_environments alias; handler/flags.rs:291–307,362–384 filters nonmatching tags. This is a forwarded backend option, not a first-class documented Elixir snapshot option. The public body map forwards it to /flags?v=2, and the affected unfiltered caller uses only documented Elixir options. This is a false absence, not reuse of another caller’s value.
Executed regression (CASE=filtered): Confirmed the context reached the HTTP seam and held the unfiltered waiter briefly to select the retry order without changing coordinator state. Base sent the unfiltered request and returned the flag; head sent no unfiltered request and failed the snapshot keys assertion ([] instead of ["deleted"] in the fixture). Consider excluding requests that filter which flags are evaluated from shared absence reads and writes, while preserving their own request body and result. Cover the context field and its wire alias, for both initial owners and retry owners, and keep the ordinary clean-sharing control.
| """ | ||
| def probe_started(name), do: GenServer.call(via(name), :probe_started, :infinity) | ||
|
|
||
| def returned_since(name, keys, started), |
There was a problem hiding this comment.
3. [P2] Retry bookkeeping can delay a completed transport error
MINOR · OPTIONAL · Introduced — lib/posthog/feature_flags/definition_loader.ex:62, RIGHT
A scoped evaluation that led a missing-key probe can have its transport error ready to return but still wait for the history GenServer. Base released the claim and returned the error without a post-error history read. Head calls returned_since/3 with :infinity to classify the round. Catching exits handles a dead process, not a busy one; the lookup also runs when the response’s own keys already settle every led key.
Executed regression (CASE=history): Both revisions returned the expected error with history running normally. Suspending the history server after entry to the HTTP stub made head remain blocked at 300 ms; base returned the error. Head returned the same error when history resumed. This isolates the new dependency, not a common production stall or a failure of the two-second waiter deadline. Consider skipping the read when direct evidence settles all led keys and making any remaining post-result classification bounded and best effort.
|
wdyt about caching three states: confirmed present, confirmed missing, unknown we already cache confirmed misses. if we also track confirmed presence, callers that need a remote result can skip the coordinated existence check and go straight to their own /flags evaluation. that way, we'd only be performing the coordination on flags whose existence is still unknown. |
|
Good call, thanks. I went with your three states instead of this PR.
I'm closing this PR. The shared retry only helps while |
💡 Motivation and Context
Follow-up to #258, as described in #258 (comment). This PR is based on #258's branch, so its diff only shows the retry work.
In #258, when the shared
/flagsrequest for a missing flag fails or reports errors, every caller that was waiting for it makes its own request at the same time. That avoids the queue that caused the incident, but it sends a burst of requests to/flagsexactly when PostHog is unhealthy.This PR adds one shared retry first. One waiting caller retries the request while the others keep waiting within their existing 2-second limit. If the retry finds out whether the flag exists, the others use that result. If it also fails or reports errors, the rest make their own requests in parallel, as in #258.
Changes
MissingProbeCoordinatornow tracks each missing key as a "round". A round is the group of callers coordinating on that key at the same time.💚 How did you test it?
mix credo --strictpass. The feature flag tests passed on repeated runs.missing_key_knowledge_test.exsadds coordinator tests for: one shared retry followed by parallel requests, late callers, callers that arrive after a failed retry, callers waiting on several keys, crashes and timeouts during the retry, nested evaluations, and the returned-key history being unavailable. Each regression test failed before its fix.errorsWhileComputingFlags, 100ms/flags/flagserrorsWhileComputingFlags, 3s/flags📝 Checklist
If releasing new changes
sampo addto generate a changeset file🤖 Agent context
Autonomy: Human-driven (agent-assisted)
main.