Repository navigation
fix(flags): stop missing-flag probes from queueing across the cluster - #258
Conversation
posthog-elixir Compliance ReportDate: 2026-11-18T11:30:12.458637+00:00 ✅ All Tests Passed!47/47 tests passed Capture Tests✅ 30/30 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
|
|
Follow-up planned: a single shared retry after an inconclusive Right now, when the shared request fails or reports errors, every waiting caller makes its own request at the same time. That avoids the queue that caused the incident. The cost is a burst of requests to I tried a single shared retry in this PR and reverted it in f9eb966. It tracked the retry per flag key instead of per group of waiting callers, which caused two bugs:
Next I'll try a correct version:
|
There was a problem hiding this comment.
PostHog Review · Standard · reviewed f9eb966: 1 Should fix.
See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.
ioannisj
left a comment
There was a problem hiding this comment.
🤖 An AI agent wrote this on behalf of @ioannisj.
No findings. The coordinator's claim, wait and release flow, the crash and call-timeout handoffs, and saving the missing-key result before waking waiters all look right to me.
💡 Motivation and Context
A customer running the SDK on a clustered BEAM app saw nodes grind to a halt right after a deploy. The cause was in
evaluate_flags.When
evaluate_flagsis called withflag_keysthat include a key missing from the local flag definitions (a deleted or non-existent flag, for example), the SDK asks/flagswhether that key exists. To avoid sending the same request many times at once, it held a lock around that request. The lock used:global.trans/2, which:So every evaluation in the cluster that asked for the same missing key waited in a single line, one
/flagsrequest at a time. The SDK remembers when the server says a key doesn't exist, which lets later calls skip the lock. That memory is per node and starts empty after a deploy. It is cleared on every definitions poll (every 30 seconds by default). It is never set when the server returns the flag, or when the response haserrorsWhileComputingFlagsorquotaLimited. In those cases every call waits in line, every time.The lock was added in #192 (v2.15.0). During that review, @ioannisj flagged this risk. The lock was kept because the evaluate-flags spec says overlapping calls for the same missing key should share one request when the server cleanly says the key doesn't exist. This PR keeps that sharing within each node, which is where the SDK remembers missing keys anyway. On
main, a 2-node cluster already sent one request per node for the same missing key, so going node-local doesn't add requests in that case.Changes
:global.translock.PostHog.FeatureFlags.MissingProbeCoordinator, a small process started under each PostHog supervisor when local evaluation is enabled. It works on one node only./flagsrequest. Other callers for that key wait for it to finish, then check again. If the server cleanly said the key doesn't exist, they skip their own request, as before./flagsreturned in the current definitions generation. If a slower request that started before the key was returned later says the key doesn't exist, that answer is ignored. Without this, a waiter that gave up and got the flag back could have its result hidden by the older request until the next refresh. A request that starts after the key was returned can still record that the key is missing, for example after the flag is deleted. If that history fills up (1,000 keys), answers from requests that started before the oldest dropped key are not recorded.The checks from the #192 review still hold:
💚 How did you test it?
mix credo --strictpass. The feature flag tests passed on repeated runs.missing_key_knowledge_test.exscover:/flagsrequest only makes another caller for the same key wait about 2 seconds, and that caller's wait is cleaned up afterwards.aand also need keybshare a single request forb./flagsendpoint. They aren't committed./flagserrorsWhileComputingFlags, 100ms/flagserrorsWhileComputingFlags, 3s/flags:peer, 20 callers, server says the flag doesn't exist📝 Checklist
If releasing new changes
sampo addto generate a changeset fileSpec divergence
The current evaluate-flags spec requires coordination but doesn't limit how long callers wait or how far coordination reaches. This PR shares requests per node, caps the wait at 2 seconds, and lets waiting callers go in parallel after an inconclusive request. A companion sdk-specs change will document that behavior.
🤖 Agent context
Autonomy: Human-driven (agent-assisted)
:global.transbut limiting it to[node()]with a retry limit. Waiters would still line up one at a time and sleep between lock attempts.