Repository navigation
fix(flags): bound missing-flag probe waits to the local node - #257
Draft
posthog[bot] wants to merge 2 commits into
Draft
posthog[bot] wants to merge 2 commits into
posthog[bot] wants to merge 2 commits into
Conversation
Concurrent evaluations of a flag with no local definition took a cluster-wide :global lock with infinite retries. The lock was held for the whole /flags request. A slow or repeated probe could stall every evaluation of that key on every node. Probe claims are now node-local and are held by a monitored holder process in the instance registry. Waiters wake when the probe settles or its owner dies, and they wait at most one second. A waiter never takes over the claim. After the wait it rechecks negative knowledge and makes its own fallback, so fallbacks for returned or failed keys run concurrently. Generated-By: PostHog Desktop Task-Id: f14e76c1-f0a9-4556-8f8f-b02c2fbae128
Rename the claim helpers and the owned-keys parameter so that they match what the code does. Build the owned keys and holders with one unzip, and release holders with a plain atom. Generated-By: PostHog Desktop Task-Id: f14e76c1-f0a9-4556-8f8f-b02c2fbae128
Contributor
posthog-elixir Compliance ReportDate: 2026-11-18T04:41:46.499992+00:00
|
| Test | Status | Duration |
|---|---|---|
| Format Validation.Event Has Required Fields | ✅ | 608ms |
| Format Validation.Event Has Uuid | ✅ | 609ms |
| Format Validation.Event Has Lib Properties | ✅ | 609ms |
| Format Validation.Distinct Id Is String | ✅ | 608ms |
| Format Validation.Token Is Present | ✅ | 608ms |
| Format Validation.Custom Properties Preserved | ✅ | 608ms |
| Format Validation.Event Has Timestamp | ✅ | 608ms |
| Format Validation.Non Utc Event Timestamp Is Converted To Utc | ❌ | 609ms |
| Retry Behavior.Retries On 503 | ✅ | 5613ms |
| Retry Behavior.Does Not Retry On 400 | ✅ | 2610ms |
| Retry Behavior.Does Not Retry On 401 | ✅ | 2611ms |
| Retry Behavior.Respects Retry After Header | ✅ | 5615ms |
| Retry Behavior.Implements Backoff | ✅ | 15624ms |
| Retry Behavior.Retries On 500 | ✅ | 5614ms |
| Retry Behavior.Retries On 502 | ✅ | 5613ms |
| Retry Behavior.Retries On 504 | ✅ | 5614ms |
| Retry Behavior.Max Retries Respected | ✅ | 15611ms |
| Deduplication.Generates Unique Uuids | ✅ | 618ms |
| Deduplication.Preserves Uuid On Retry | ✅ | 5614ms |
| Deduplication.Preserves Uuid And Timestamp On Retry | ✅ | 10618ms |
| Deduplication.Preserves Uuid And Timestamp On Batch Retry | ✅ | 5616ms |
| Deduplication.No Duplicate Events In Batch | ✅ | 613ms |
| Deduplication.Different Events Have Different Uuids | ✅ | 611ms |
| Compression.Sends Gzip When Enabled | ✅ | 608ms |
| Batch Format.Uses Proper Batch Structure | ✅ | 608ms |
| Batch Format.Flush With No Events Sends Nothing | ✅ | 606ms |
| Batch Format.Multiple Events Batched Together | ✅ | 612ms |
| Error Handling.Does Not Retry On 403 | ✅ | 2610ms |
| Error Handling.Does Not Retry On 413 | ✅ | 2611ms |
| Error Handling.Retries On 408 | ✅ | 5615ms |
Failures
format_validation.non_utc_event_timestamp_is_converted_to_utc
Event 0 field 'timestamp' instant '2026-11-18T04:40:02.499895Z' != expected '2025-01-02T03:04:05Z'
Feature_Flags Tests
View Details
| Test | Status | Duration |
|---|---|---|
| Request Payload.Request With Person Properties Device Id | ✅ | 7ms |
| Request Payload.Flags Request Uses V2 Query Param | ✅ | 6ms |
| Request Payload.Flags Request Hits Flags Path Not Decide | ✅ | 5ms |
| Request Payload.Flags Request Omits Authorization Header | ✅ | 5ms |
| Request Payload.Token In Flags Body Matches Init | ✅ | 6ms |
| Request Payload.Groups Round Trip | ✅ | 5ms |
| Request Payload.Groups Default To Empty Object | ❌ | 6ms |
| Request Payload.Disable Geoip False Propagates As Geoip Disable False | ✅ | 5ms |
| Request Payload.Disable Geoip Omitted Defaults To False | ❌ | 5ms |
| Request Payload.Flag Keys To Evaluate Contains Only Requested Key | ✅ | 5ms |
| Request Lifecycle.No Flags Request On Init Alone | ✅ | 3ms |
| Request Lifecycle.No Flags Request On Normal Capture | ✅ | 608ms |
| Request Lifecycle.Two Flag Calls Produce Two Remote Requests | ✅ | 10ms |
| Request Lifecycle.Mock Response Value Is Returned To Caller | ❌ | 6ms |
| Retry Behavior.Retries Flags On 502 | ❌ | 309ms |
| Retry Behavior.Retries Flags On 504 | ❌ | 309ms |
| Side Effect Events.Get Feature Flag Captures Feature Flag Called Event | ❌ | 609ms |
Failures
request_payload.groups_default_to_empty_object
Field 'groups' not found in /flags request body at path 'groups'. Available keys: ['distinct_id', 'api_key', 'flag_keys_to_evaluate']
request_payload.disable_geoip_omitted_defaults_to_false
Field 'geoip_disable' not found in /flags request body at path 'geoip_disable'. Available keys: ['distinct_id', 'api_key', 'flag_keys_to_evaluate']
request_lifecycle.mock_response_value_is_returned_to_caller
Last action result missing field 'value'. Keys: ['error', 'success']
retry_behavior.retries_flags_on_502
Last action result missing field 'value'. Keys: ['error', 'success']
retry_behavior.retries_flags_on_504
Last action result missing field 'value'. Keys: ['error', 'success']
side_effect_events.get_feature_flag_captures_feature_flag_called_event
Expected 1 events with name '$feature_flag_called', got 0
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 Motivation and Context
Problem
with_missing_probe_locksuses:global.transwith infinite retries. It holds the lock for the full remote/flagsrequest, retries included. Every evaluation of that key on every node waits behind that request.:globalretries sleep for a random time that grows to 8 s. Each retry also goes throughglobal_name_serveron every node./flagsrequest while it holds the lock. This makes the requests run one at a time across the cluster.Changes
Registry. Waiters monitor the holder, so they wake as soon as the probe settles or its owner dies.:global)Spec divergence (evaluate-flags)
Note
The wait bound is an internal constant (
@missing_probe_wait_ms), not a new option. The SDK has no:telemetrydependency, so this PR adds no wait-time metric.💚 How did you test it?
mix test→ 1049 tests, 0 failures.mix format --check-formattedandmix credo --strictpass.missing_key_knowledge_test.exs:main: the waiter never continues.main: the requests run one at a time.missing_key_knowledge_test.exs5 times to check for timing flakiness. All runs passed.📝 Checklist
If releasing new changes
sampo addto generate a changeset file🤖 Agent context
Autonomy: Fully autonomous
:global.trans(lock, fun, [node()], retries). It still uses random backoff, so it gives no exact time bound, and it goes throughglobal_name_server.Registry, and owner-side waiter lists. Owner-side lists can leave stray messages in a caller's mailbox, for example a GenServer. A monitored holder process gives exact wake-ups and does not affect the caller's mailbox.sampo addformat, because thesampoCLI is not installed in the sandbox.Created with PostHog Desktop from this inbox report.
Origin