Skip to content

fix(flags): stop missing-flag probes from queueing across the cluster - #258

Merged
marandaneto merged 14 commits into
mainfrom
fix/flags-missing-probe-lock
Nov 18, 2026
Merged

marandaneto merged 14 commits into
mainfrom
fix/flags-missing-probe-lock

Conversation

@marandaneto

@marandaneto marandaneto commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

💡 Motivation and Context

A customer running the SDK on a clustered BEAM app saw nodes grind to a halt right after a deploy. The cause was in evaluate_flags.

When evaluate_flags is called with flag_keys that include a key missing from the local flag definitions (a deleted or non-existent flag, for example), the SDK asks /flags whether that key exists. To avoid sending the same request many times at once, it held a lock around that request. The lock used :global.trans/2, which:

  • locks across every connected node in the cluster, not just the local node.
  • retries forever, sleeping a random time of up to 8 seconds between attempts, with no overall timeout.
  • stays held for the whole HTTP request.

So every evaluation in the cluster that asked for the same missing key waited in a single line, one /flags request at a time. The SDK remembers when the server says a key doesn't exist, which lets later calls skip the lock. That memory is per node and starts empty after a deploy. It is cleared on every definitions poll (every 30 seconds by default). It is never set when the server returns the flag, or when the response has errorsWhileComputingFlags or quotaLimited. In those cases every call waits in line, every time.

The lock was added in #192 (v2.15.0). During that review, @ioannisj flagged this risk. The lock was kept because the evaluate-flags spec says overlapping calls for the same missing key should share one request when the server cleanly says the key doesn't exist. This PR keeps that sharing within each node, which is where the SDK remembers missing keys anyway. On main, a 2-node cluster already sent one request per node for the same missing key, so going node-local doesn't add requests in that case.

Changes

  • Removes the :global.trans lock.
  • Adds PostHog.FeatureFlags.MissingProbeCoordinator, a small process started under each PostHog supervisor when local evaluation is enabled. It works on one node only.
    • The first caller for a missing key makes the /flags request. Other callers for that key wait for it to finish, then check again. If the server cleanly said the key doesn't exist, they skip their own request, as before.
    • One deadline per call caps the total wait at 2 seconds, even when the call waits on several keys. After that, the caller makes its own request.
    • When the request returns the flag, fails, or reports errors, every waiting caller makes its own request at the same time instead of one after another.
    • A caller that is waiting doesn't block other keys, so two callers can never wait on each other.
    • If the caller making the request crashes or raises before finishing, one waiting caller takes over the request. A waiting caller that is killed is removed from the coordinator's waiter list.
    • If the coordinator doesn't answer in time, the caller makes its own request.
  • The SDK now remembers which keys /flags returned in the current definitions generation. If a slower request that started before the key was returned later says the key doesn't exist, that answer is ignored. Without this, a waiter that gave up and got the flag back could have its result hidden by the older request until the next refresh. A request that starts after the key was returned can still record that the key is missing, for example after the flag is deleted. If that history fills up (1,000 keys), answers from requests that started before the oldest dropped key are not recorded.

The checks from the #192 review still hold:

  • A caller only skips its request after the SDK has saved the "missing" result. Saving happens before the first caller releases the key.
  • Only the keys a caller was assigned to check are saved as missing. A delayed response still can't re-add a key that a newer response returned.
  • Requests for different keys still run in parallel.

💚 How did you test it?

  • The full test suite and mix credo --strict pass. The feature flag tests passed on repeated runs.
  • New tests in missing_key_knowledge_test.exs cover:
    • A stalled /flags request only makes another caller for the same key wait about 2 seconds, and that caller's wait is cleaned up afterwards.
    • Callers that waited on key a and also need key b share a single request for b.
    • When the shared request returns the flag or reports errors, the waiting callers make their requests at the same time.
    • When the caller making the request is killed or raises, exactly one waiting caller takes over.
    • A killed waiting caller is removed from the coordinator.
    • A slower request's "doesn't exist" answer can't hide a key that a newer request returned.
    • Returned-key history overflow only affects requests that overlapped it.
  • The existing tests for missing-key handling still pass unchanged.
  • I also ran throwaway reproduction scripts with a mocked /flags endpoint. They aren't committed.
Scenario Before After
50 callers, server returns the flag, 100ms /flags 11.4s, 1 request at a time 0.2s, 49 at a time
50 callers, errorsWhileComputingFlags, 100ms /flags 13.1s, 1 at a time 0.2s, 49 at a time
50 callers, errorsWhileComputingFlags, 3s /flags about 150s 5s (2s wait, then 3s request)
50 callers, server says the flag doesn't exist 1 request 1 request, 0.1s
2-node cluster using :peer, 20 callers, server says the flag doesn't exist 1.03s, 1 request per node 0.22s, 1 request per node

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

Spec divergence

The current evaluate-flags spec requires coordination but doesn't limit how long callers wait or how far coordination reaches. This PR shares requests per node, caps the wait at 2 seconds, and lets waiting callers go in parallel after an inconclusive request. A companion sdk-specs change will document that behavior.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

  • Investigated and implemented with Pi (Claude). Several autoreview passes with GPT found real issues: waiters that didn't share follow-up requests, stale waiter entries, failed requests reported as finished, and slower requests hiding returned keys. Each fix has a regression test.
  • A shared retry after an errored request was tried, then reverted in the last commit. It was tracked per key instead of per group of waiting callers, so late callers could miss its result and new callers could skip coordination. A correct version may follow separately.
  • Rejected option: keeping :global.trans but limiting it to [node()] with a retry limit. Waiters would still line up one at a time and sleep between lock attempts.

@marandaneto marandaneto self-assigned this Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

posthog-elixir Compliance Report

Date: 2026-11-18T11:30:12.458637+00:00
Duration: 111217ms

✅ All Tests Passed!

47/47 tests passed


Capture Tests

✅ 30/30 tests passed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 610ms
Format Validation.Event Has Uuid ✅ 610ms
Format Validation.Event Has Lib Properties ✅ 610ms
Format Validation.Distinct Id Is String ✅ 610ms
Format Validation.Token Is Present ✅ 610ms
Format Validation.Custom Properties Preserved ✅ 610ms
Format Validation.Event Has Timestamp ✅ 610ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ✅ 610ms
Retry Behavior.Retries On 503 ✅ 5614ms
Retry Behavior.Does Not Retry On 400 ✅ 2612ms
Retry Behavior.Does Not Retry On 401 ✅ 2613ms
Retry Behavior.Respects Retry After Header ✅ 5616ms
Retry Behavior.Implements Backoff ✅ 15623ms
Retry Behavior.Retries On 500 ✅ 5615ms
Retry Behavior.Retries On 502 ✅ 5615ms
Retry Behavior.Retries On 504 ✅ 5615ms
Retry Behavior.Max Retries Respected ✅ 15611ms
Deduplication.Generates Unique Uuids ✅ 621ms
Deduplication.Preserves Uuid On Retry ✅ 5613ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 10614ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5616ms
Deduplication.No Duplicate Events In Batch ✅ 616ms
Deduplication.Different Events Have Different Uuids ✅ 613ms
Compression.Sends Gzip When Enabled ✅ 610ms
Batch Format.Uses Proper Batch Structure ✅ 611ms
Batch Format.Flush With No Events Sends Nothing ✅ 607ms
Batch Format.Multiple Events Batched Together ✅ 615ms
Error Handling.Does Not Retry On 403 ✅ 2611ms
Error Handling.Does Not Retry On 413 ✅ 2612ms
Error Handling.Retries On 408 ✅ 5616ms

Feature_Flags Tests

✅ 17/17 tests passed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 9ms
Request Payload.Flags Request Uses V2 Query Param ✅ 6ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 7ms
Request Payload.Flags Request Omits Authorization Header ✅ 6ms
Request Payload.Token In Flags Body Matches Init ✅ 7ms
Request Payload.Groups Round Trip ✅ 6ms
Request Payload.Groups Default To Empty Object ✅ 7ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 6ms
Request Payload.Disable Geoip Omitted Defaults To False ✅ 6ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 7ms
Request Lifecycle.No Flags Request On Init Alone ✅ 3ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 608ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 11ms
Request Lifecycle.Mock Response Value Is Returned To Caller ✅ 609ms
Retry Behavior.Retries Flags On 502 ✅ 912ms
Retry Behavior.Retries Flags On 504 ✅ 912ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ✅ 1213ms

@marandaneto

Copy link
Copy Markdown
Member Author

Follow-up planned: a single shared retry after an inconclusive /flags request.

Right now, when the shared request fails or reports errors, every waiting caller makes its own request at the same time. That avoids the queue that caused the incident. The cost is a burst of requests to /flags while PostHog is returning errors.

I tried a single shared retry in this PR and reverted it in f9eb966. It tracked the retry per flag key instead of per group of waiting callers, which caused two bugs:

  • A caller that started waiting late could miss a retry that succeeded.
  • A brand-new caller could see a retry that already failed and skip coordination entirely.

Next I'll try a correct version:

  • Track the retry per group of callers that were waiting on the failed request. Callers that arrive later start fresh coordination and don't inherit that group's retry result.
  • One caller from the group makes the retry while the others keep waiting within their existing 2-second deadline. If the retry also fails, the rest of the group makes its own requests in parallel. There is no second retry for the same group.
  • Add a regression test for each bug above, rerun autoreview, then push and update this description.

@marandaneto
marandaneto marked this pull request as ready for review November 18, 2026 08:37
@marandaneto
marandaneto requested a review from a team as a code owner November 18, 2026 08:37
@veria-ai

veria-ai Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

PR overview

No security issues were found in this pull request.

Reviews

@posthog posthog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed f9eb966: 1 Should fix.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

@ioannisj ioannisj left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 An AI agent wrote this on behalf of @ioannisj.

No findings. The coordinator's claim, wait and release flow, the crash and call-timeout handoffs, and saving the missing-key result before waking waiters all look right to me.

@marandaneto
marandaneto merged commit c82105e into main Nov 18, 2026
30 checks passed
@marandaneto
marandaneto deleted the fix/flags-missing-probe-lock branch November 18, 2026 13:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants