Skip to content

fix(flags): give waiting callers one shared retry for inconclusive missing-flag requests - #259

Closed
marandaneto wants to merge 15 commits into
mainfrom
fix/flags-probe-shared-retry
Closed

marandaneto wants to merge 15 commits into
mainfrom
fix/flags-probe-shared-retry

Conversation

@marandaneto

Copy link
Copy Markdown
Member

💡 Motivation and Context

Follow-up to #258, as described in #258 (comment). This PR is based on #258's branch, so its diff only shows the retry work.

In #258, when the shared /flags request for a missing flag fails or reports errors, every caller that was waiting for it makes its own request at the same time. That avoids the queue that caused the incident, but it sends a burst of requests to /flags exactly when PostHog is unhealthy.

This PR adds one shared retry first. One waiting caller retries the request while the others keep waiting within their existing 2-second limit. If the retry finds out whether the flag exists, the others use that result. If it also fails or reports errors, the rest make their own requests in parallel, as in #258.

Changes

MissingProbeCoordinator now tracks each missing key as a "round". A round is the group of callers coordinating on that key at the same time.

  • Each caller remembers the round it joined. A round ends when a request establishes whether the flag exists, or when its one shared retry is also inconclusive.
  • A caller holding an ended round skips that key and makes its own request. It never joins a newer round or a later caller's request.
  • Callers that arrive after a round ends start a new round, with its own shared retry.
  • A round only keeps its shared retry while some caller can still use it. That includes callers waiting on a different key in the same call. A failed request with no other callers ends its round, so it can't use up the next group's retry.
  • If the caller making the request crashes or raises, the round reopens for the next caller without using up the shared retry.
  • Calls are tracked individually, so an evaluation nested inside another one can't release or take over the outer call's request.
  • The SDK checks which keys the response itself returned instead of relying only on the bounded returned-key history. Those keys always count as settled.

💚 How did you test it?

  • The full test suite and mix credo --strict pass. The feature flag tests passed on repeated runs.
  • missing_key_knowledge_test.exs adds coordinator tests for: one shared retry followed by parallel requests, late callers, callers that arrive after a failed retry, callers waiting on several keys, crashes and timeouts during the retry, nested evaluations, and the returned-key history being unavailable. Each regression test failed before its fix.
  • I reran the same throwaway load scripts as fix(flags): stop missing-flag probes from queueing across the cluster #258:
Scenario #258 This PR
50 callers, errorsWhileComputingFlags, 100ms /flags 0.2s, 49 at a time 0.3s, one shared retry, then 48 at a time
50 callers, server returns the flag, 100ms /flags 0.2s, 49 at a time 0.2s, 49 at a time
50 callers, errorsWhileComputingFlags, 3s /flags 5s 5s (each caller's 2s wait limit still applies)
50 callers, server says the flag doesn't exist 1 request 1 request
2-node cluster, 20 callers, server says the flag doesn't exist 1 request per node 1 request per node

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

@marandaneto marandaneto self-assigned this Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

posthog-elixir Compliance Report

Date: 2026-11-18T13:14:17.499773+00:00
Duration: 111196ms

✅ All Tests Passed!

47/47 tests passed


Capture Tests

✅ 30/30 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 609ms
Format Validation.Event Has Uuid ✅ 609ms
Format Validation.Event Has Lib Properties ✅ 609ms
Format Validation.Distinct Id Is String ✅ 609ms
Format Validation.Token Is Present ✅ 608ms
Format Validation.Custom Properties Preserved ✅ 609ms
Format Validation.Event Has Timestamp ✅ 609ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 609ms
Retry Behavior.Retries On 503 ✅ 5612ms
Retry Behavior.Does Not Retry On 400 ✅ 2610ms
Retry Behavior.Does Not Retry On 401 ✅ 2611ms
Retry Behavior.Respects Retry After Header ✅ 5613ms
Retry Behavior.Implements Backoff ✅ 15625ms
Retry Behavior.Retries On 500 ✅ 5614ms
Retry Behavior.Retries On 502 ✅ 5615ms
Retry Behavior.Retries On 504 ✅ 5615ms
Retry Behavior.Max Retries Respected ✅ 15617ms
Deduplication.Generates Unique Uuids ✅ 619ms
Deduplication.Preserves Uuid On Retry ✅ 5613ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 10617ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5616ms
Deduplication.No Duplicate Events In Batch ✅ 613ms
Deduplication.Different Events Have Different Uuids ✅ 611ms
Compression.Sends Gzip When Enabled ✅ 608ms
Batch Format.Uses Proper Batch Structure ✅ 609ms
Batch Format.Flush With No Events Sends Nothing ✅ 606ms
Batch Format.Multiple Events Batched Together ✅ 613ms
Error Handling.Does Not Retry On 403 ✅ 2611ms
Error Handling.Does Not Retry On 413 ✅ 2611ms
Error Handling.Retries On 408 ✅ 5614ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 8ms
Request Payload.Flags Request Uses V2 Query Param ✅ 6ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 5ms
Request Payload.Flags Request Omits Authorization Header ✅ 5ms
Request Payload.Token In Flags Body Matches Init ✅ 6ms
Request Payload.Groups Round Trip ✅ 6ms
Request Payload.Groups Default To Empty Object ✅ 6ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 6ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 5ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 6ms
Request Lifecycle.No Flags Request On Init Alone ✅ 3ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 608ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 10ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 607ms
Retry Behavior.Retries Flags On 502 ✅ 911ms
Retry Behavior.Retries Flags On 504 ✅ 911ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 1212ms

Base automatically changed from fix/flags-missing-probe-lock to main November 18, 2026 13:04
Each key has at most one coordination round with a single shared retry. Callers keep the round id they joined, so they skip a key once its round ends instead of joining a later probe. Releases and give-ups are scoped to the coordinated call that made them.

@dustinbyrne dustinbyrne left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified that a clean shared retry can avoid a second waiter’s request. Two regressions need correction before approval: bulk release does quadratic work in the coordinator, and a context-filtered retry can make an unfiltered caller miss an existing flag. Both reproduced through evaluate_flags/2 on this head and passed on base. Details and suggested fixes are inline.

AI-assisted review.

end

defp maybe_forget(state, pid) do
leads? = Enum.any?(state.owners, fn {_key, owner} -> owner == pid end)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1. [P1] Bulk probe release does quadratic work on the coordinator

IMPORTANT · REQUIRED · Introduced — lib/posthog/feature_flags/missing_probe_coordinator.ex:387, RIGHT

When many FeatureFlags.evaluate_flags(name, %{distinct_id: id, flag_keys: keys}) calls overlap on a bulk scope missing from loaded local definitions, even a clean, prompt /flags response can leave the instance’s coordinator busy for hundreds of milliseconds. Releasing a probe should cost roughly the work of notifying its waiters, not scanning all active owners for each notification.

notify_waiters/3 calls maybe_forget/2 once per waiter per key. Head scans state.owners before checking whether the waiter still has an active wait or round membership. With K owned keys and W waiters who own none, that visits about W × K × (K−1) / 2 entries. Base checked the caller’s own owned set instead.

Executed regression (CASE=perf): Gate one clean omission until 50 waiters have registered, then drain the group. At 1,000 keys, base used 1.39M coordinator reductions / 44 ms; head used 227.25M / 496 ms. Doubling the scope from 500 to 1,000 keys roughly doubled base work and nearly quadrupled head work, so at_1000 < 3 * at_500 passed on base and failed on head. Both revisions made one remote request and returned the expected empty snapshots. A disjoint control also showed about 37K → 1.52M reductions for 50 one-key calls with 1,000 unrelated owners live; that control includes membership cleanup as well as ownership scans.

The 1,000-key scope is within the supported coordination limit. This is a stress case, not a production latency forecast; no deadline miss was measured. The coordinator processes the release serially, so unrelated calls can queue behind it. Consider checking wait and membership state first, restoring process-local owned-key tracking for cleanup, and skipping membership cleanup for claims that never joined a round. Keep the reductions test and the one-request clean-omission assertion.

{remote_results, malformed?} = build_remote_results(scoped_remote_flags, response_body)
put_probe_returned(Map.keys(scoped_remote_flags))

update_negative_knowledge(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2. [P1] A context-filtered retry can hide an existing flag from an unfiltered caller

IMPORTANT · REQUIRED · Amplifies a pre-existing omission classifier — lib/posthog/feature_flags.ex:630, RIGHT

Suppose loaded local definitions are stale and lack a flag tagged production. An initial probe fails, and these two callers are waiting on its key:

# In separate caller processes, with local evaluation enabled:
PostHog.FeatureFlags.evaluate_flags(name, %{
  distinct_id: "staging-user",
  flag_keys: ["checkout"],
  evaluation_contexts: ["staging"]
})

PostHog.FeatureFlags.evaluate_flags(name, %{
  distinct_id: "normal-user",
  flag_keys: ["checkout"]
})

If the staging caller leads the shared retry, its clean response legitimately omits checkout for that request. Head can record that omission as instance-wide absence. The normal caller then skips /flags and gets an empty snapshot instead of its enabled flag; later unfiltered callers may also miss it until refresh, eviction, or a positive response clears it. Base released the staging waiter without ownership after the first error, so its response could not write that absence. The initial-owner version of the classifier bug already existed; the retry creates a new path to it.

The official /flags API documents evaluation_contexts. At backend 036a0ced6d1ef86987ea28f692c8161c26586967, flags/flag_request.rs:100–101 accepts it and the evaluation_environments alias; handler/flags.rs:291–307,362–384 filters nonmatching tags. This is a forwarded backend option, not a first-class documented Elixir snapshot option. The public body map forwards it to /flags?v=2, and the affected unfiltered caller uses only documented Elixir options. This is a false absence, not reuse of another caller’s value.

Executed regression (CASE=filtered): Confirmed the context reached the HTTP seam and held the unfiltered waiter briefly to select the retry order without changing coordinator state. Base sent the unfiltered request and returned the flag; head sent no unfiltered request and failed the snapshot keys assertion ([] instead of ["deleted"] in the fixture). Consider excluding requests that filter which flags are evaluated from shared absence reads and writes, while preserving their own request body and result. Cover the context field and its wire alias, for both initial owners and retry owners, and keep the ordinary clean-sharing control.

"""
def probe_started(name), do: GenServer.call(via(name), :probe_started, :infinity)

def returned_since(name, keys, started),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3. [P2] Retry bookkeeping can delay a completed transport error

MINOR · OPTIONAL · Introduced — lib/posthog/feature_flags/definition_loader.ex:62, RIGHT

A scoped evaluation that led a missing-key probe can have its transport error ready to return but still wait for the history GenServer. Base released the claim and returned the error without a post-error history read. Head calls returned_since/3 with :infinity to classify the round. Catching exits handles a dead process, not a busy one; the lookup also runs when the response’s own keys already settle every led key.

Executed regression (CASE=history): Both revisions returned the expected error with history running normally. Suspending the history server after entry to the HTTP stub made head remain blocked at 300 ms; base returned the error. Head returned the same error when history resumed. This isolates the new dependency, not a common production stall or a failure of the two-second waiter deadline. Consider skipping the read when direct evidence settles all led keys and making any remaining post-result classification bounded and best effort.

@dustinbyrne

Copy link
Copy Markdown
Contributor

wdyt about caching three states: confirmed present, confirmed missing, unknown

we already cache confirmed misses. if we also track confirmed presence, callers that need a remote result can skip the coordinated existence check and go straight to their own /flags evaluation. that way, we'd only be performing the coordination on flags whose existence is still unknown.

@marandaneto

Copy link
Copy Markdown
Member Author

Good call, thanks. I went with your three states instead of this PR.

I'm closing this PR. The shared retry only helps while /flags returns errors, adds about 1,200 lines of coordinator state, and the burst it prevents is limited to one request per waiting caller. The branch stays around in case error bursts turn out to matter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants