Skip to content

fix(flags): bound missing-flag probe waits to the local node - #257

Draft
posthog[bot] wants to merge 2 commits into
mainfrom
posthog-self-driving/fixflags-stop-serializing-missing-flag-f619dc
Draft

posthog[bot] wants to merge 2 commits into
mainfrom
posthog-self-driving/fixflags-stop-serializing-missing-flag-f619dc

Conversation

@posthog

@posthog posthog Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

💡 Motivation and Context

Problem

  • A flag check can stall the whole BEAM cluster when the flag has no local definition. Examples are a deleted flag, or a flag that is only in a different project.
  • with_missing_probe_locks uses :global.trans with infinite retries. It holds the lock for the full remote /flags request, retries included. Every evaluation of that key on every node waits behind that request.
  • :global retries sleep for a random time that grows to 8 s. Each retry also goes through global_name_server on every node.
  • After a probe returns the key, or fails, each waiter gets the lock in turn and makes its own /flags request while it holds the lock. This makes the requests run one at a time across the cluster.
  • This behavior is in all releases since v2.15.0 (feat(flags): add local feature flag evaluation #192).

Changes

  • The probe claim is now node-local and single-flight. A small holder process keeps the claim in the instance's existing Registry. Waiters monitor the holder, so they wake as soon as the probe settles or its owner dies.
  • The wait is bounded: one deadline of 1 s for all of a call's missing keys. After the deadline, the waiter makes its own fallback request.
  • A waiter never takes over the claim. After the wait, it rechecks the missing-key knowledge. If the key is still unresolved, it makes its own fallback without a lock. Only keys that a caller claims can set negative knowledge from its response.
Case Before After
Lock scope Cluster (:global) One node, one SDK instance
Max wait No limit 1 s
Wake-up Random backoff, up to 8 s When the holder exits
Waiters after a returned or failed probe One at a time, with the lock held All at the same time, no lock
Clean omission Coalesced Coalesced (no change)

Spec divergence (evaluate-flags)

  • The spec says that overlapping evaluations coordinate while a probe is in flight. A waiter that times out after 1 s now makes its own request. This can cause a duplicate request when a probe takes more than 1 s. We think liveness is more important than deduplication in this case.
  • Coordination is per node, not per cluster. The spec does not require cross-node coordination.
  • The report also asked us to keep known-missing keys across definition reloads. This PR does not do that. The spec says that knowledge "SHALL be cleared after every successful local-definition refresh". With a bounded wait, one probe per node per key per refresh no longer causes a queue.

Note

The wait bound is an internal constant (@missing_probe_wait_ms), not a new option. The SDK has no :telemetry dependency, so this PR adds no wait-time metric.

💚 How did you test it?

  • Ran the tests on Elixir 1.18.3 / OTP 27.3 (same as CI): mix test → 1049 tests, 0 failures. mix format --check-formatted and mix credo --strict pass.
  • New tests in missing_key_knowledge_test.exs:
    • A waiter behind a stalled probe makes its own fallback after the bound. This test fails on main: the waiter never continues.
    • Waiters behind a probe that returns the key make their requests at the same time. This test fails on main: the requests run one at a time.
    • When the probe owner is killed, its waiter continues immediately, not after the 1 s bound.
  • The existing coalescing, mixed-set, disjoint-set, generation, and ordering tests pass without changes.
  • Ran missing_key_knowledge_test.exs 5 times to check for timing flakiness. All runs passed.
  • Not tested: multi-node behavior in a real cluster.

📝 Checklist

  • All commits are signed (see Commit signing)
  • I reviewed the submitted code.
  • I added tests to verify the changes.
  • I updated the docs if needed.
  • No breaking change or entry added to the changelog.

If releasing new changes

  • Ran sampo add to generate a changeset file

🤖 Agent context

Autonomy: Fully autonomous

  • PostHog Desktop agent (Claude) wrote this from an inbox report. The report linked the missing-flag lock to a production stall.
  • Rejected :global.trans(lock, fun, [node()], retries). It still uses random backoff, so it gives no exact time bound, and it goes through global_name_server.
  • Rejected polling the Registry, and owner-side waiter lists. Owner-side lists can leave stray messages in a caller's mailbox, for example a GenServer. A monitored holder process gives exact wake-ups and does not affect the caller's mailbox.
  • The changeset file was written by hand in the sampo add format, because the sampo CLI is not installed in the sandbox.

Created with PostHog Desktop from this inbox report.

Origin

  • Conversations: ticket
  • First signal: 2026-11-18
  • Inbox report: open
  • Task started by: auto-start, after the report was rated P0 and ready to fix

Concurrent evaluations of a flag with no local definition took a cluster-wide :global lock with infinite retries. The lock was held for the whole /flags request. A slow or repeated probe could stall every evaluation of that key on every node.

Probe claims are now node-local and are held by a monitored holder process in the instance registry. Waiters wake when the probe settles or its owner dies, and they wait at most one second. A waiter never takes over the claim. After the wait it rechecks negative knowledge and makes its own fallback, so fallbacks for returned or failed keys run concurrently.

Generated-By: PostHog Desktop
Task-Id: f14e76c1-f0a9-4556-8f8f-b02c2fbae128
@posthog posthog Bot added the self-driving label Nov 18, 2026
@posthog
posthog Bot requested a review from marandaneto November 18, 2026 04:37
Rename the claim helpers and the owned-keys parameter so that they match what the code does. Build the owned keys and holders with one unzip, and release holders with a plain atom.

Generated-By: PostHog Desktop
Task-Id: f14e76c1-f0a9-4556-8f8f-b02c2fbae128
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

posthog-elixir Compliance Report

Date: 2026-11-18T04:41:46.499992+00:00
Duration: 108474ms

⚠️ Some Tests Failed

40/47 tests passed, 7 failed


Capture Tests

⚠️ 29/30 tests passed, 1 failed

View Details
Test Status Duration
Format Validation.Event Has Required Fields ✅ 608ms
Format Validation.Event Has Uuid ✅ 609ms
Format Validation.Event Has Lib Properties ✅ 609ms
Format Validation.Distinct Id Is String ✅ 608ms
Format Validation.Token Is Present ✅ 608ms
Format Validation.Custom Properties Preserved ✅ 608ms
Format Validation.Event Has Timestamp ✅ 608ms
Format Validation.Non Utc Event Timestamp Is Converted To Utc ❌ 609ms
Retry Behavior.Retries On 503 ✅ 5613ms
Retry Behavior.Does Not Retry On 400 ✅ 2610ms
Retry Behavior.Does Not Retry On 401 ✅ 2611ms
Retry Behavior.Respects Retry After Header ✅ 5615ms
Retry Behavior.Implements Backoff ✅ 15624ms
Retry Behavior.Retries On 500 ✅ 5614ms
Retry Behavior.Retries On 502 ✅ 5613ms
Retry Behavior.Retries On 504 ✅ 5614ms
Retry Behavior.Max Retries Respected ✅ 15611ms
Deduplication.Generates Unique Uuids ✅ 618ms
Deduplication.Preserves Uuid On Retry ✅ 5614ms
Deduplication.Preserves Uuid And Timestamp On Retry ✅ 10618ms
Deduplication.Preserves Uuid And Timestamp On Batch Retry ✅ 5616ms
Deduplication.No Duplicate Events In Batch ✅ 613ms
Deduplication.Different Events Have Different Uuids ✅ 611ms
Compression.Sends Gzip When Enabled ✅ 608ms
Batch Format.Uses Proper Batch Structure ✅ 608ms
Batch Format.Flush With No Events Sends Nothing ✅ 606ms
Batch Format.Multiple Events Batched Together ✅ 612ms
Error Handling.Does Not Retry On 403 ✅ 2610ms
Error Handling.Does Not Retry On 413 ✅ 2611ms
Error Handling.Retries On 408 ✅ 5615ms

Failures

format_validation.non_utc_event_timestamp_is_converted_to_utc

Event 0 field 'timestamp' instant '2026-11-18T04:40:02.499895Z' != expected '2025-01-02T03:04:05Z'

Feature_Flags Tests

⚠️ 11/17 tests passed, 6 failed

View Details
Test Status Duration
Request Payload.Request With Person Properties Device Id ✅ 7ms
Request Payload.Flags Request Uses V2 Query Param ✅ 6ms
Request Payload.Flags Request Hits Flags Path Not Decide ✅ 5ms
Request Payload.Flags Request Omits Authorization Header ✅ 5ms
Request Payload.Token In Flags Body Matches Init ✅ 6ms
Request Payload.Groups Round Trip ✅ 5ms
Request Payload.Groups Default To Empty Object ❌ 6ms
Request Payload.Disable Geoip False Propagates As Geoip Disable False ✅ 5ms
Request Payload.Disable Geoip Omitted Defaults To False ❌ 5ms
Request Payload.Flag Keys To Evaluate Contains Only Requested Key ✅ 5ms
Request Lifecycle.No Flags Request On Init Alone ✅ 3ms
Request Lifecycle.No Flags Request On Normal Capture ✅ 608ms
Request Lifecycle.Two Flag Calls Produce Two Remote Requests ✅ 10ms
Request Lifecycle.Mock Response Value Is Returned To Caller ❌ 6ms
Retry Behavior.Retries Flags On 502 ❌ 309ms
Retry Behavior.Retries Flags On 504 ❌ 309ms
Side Effect Events.Get Feature Flag Captures Feature Flag Called Event ❌ 609ms

Failures

request_payload.groups_default_to_empty_object

Field 'groups' not found in /flags request body at path 'groups'. Available keys: ['distinct_id', 'api_key', 'flag_keys_to_evaluate']

request_payload.disable_geoip_omitted_defaults_to_false

Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'api_key', 'flag_keys_to_evaluate']

request_lifecycle.mock_response_value_is_returned_to_caller

Last action result missing field 'value'. Keys: ['error', 'success']

retry_behavior.retries_flags_on_502

Last action result missing field 'value'. Keys: ['error', 'success']

retry_behavior.retries_flags_on_504

Last action result missing field 'value'. Keys: ['error', 'success']

side_effect_events.get_feature_flag_captures_feature_flag_called_event

Expected 1 events with name '$feature_flag_called', got 0

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants