Repository navigation
fix(migrate-email): leave send retries to PostHog's SDK - #541
Conversation
The SDKs retry, with the same idempotency key, only the answers that mean PostHog didn't take the send. The skill now removes an app's own retry around the send instead of keeping or rebuilding it, moves .NET SendGrid sites whose client retried, and keeps the SDK's retries for a deadline site that retried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e .NET throw A deadline send in Go or .NET whose request retried no longer takes the rewrite file's zero retries. A .NET SendGrid client that retried threw once its last retry failed, so its moved site throws PostHog's error for a server error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only SendGrid's own retry handler threw once its retries ran out; a retry handler on the HttpClient keeps what it did. A deadline the app set for each try becomes the email client's timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job framework's rerun is left alone, as on the provider; only an idempotency key the site sets keeps a rerun from sending twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
PostHog Review · Standard · reviewed 1ba742d: 1 Should fix.
See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.
… can A retry loop that shares one deadline across its tries, such as one AbortSignal over every fetch, keeps that deadline only where the send takes it (Go's context, .NET's token) or a timer stays around the send. Elsewhere the SDK can't keep it, and the site stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'t keep stays Name the case in the stay sentence and say such a deadline is never the client's timeout, with p-retry maxRetryTime and tenacity stop_after_delay as examples. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A timer around a send whose SDK keeps retrying rejects while the send can still deliver, so only a send that takes the deadline keeps it. State the stay rule once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n .NET too A deadline set for each try is the email client's timeout, so its timeout comes back as network_error, never as the token's OperationCanceledException that the .NET bullet describes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uage bullets Say in the exception's lead-in that a deadline set for each try is never one the send takes, so an agent never applies the Go or .NET bullet to it. The last bullet goes back to its base wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| - In every other SDK the deadline is the client's timeout, and its timeout is a `network_error` like any other network failure: a branch on it stays. | ||
| - A branch on the provider's answer to an invalid send, such as a `400` or a `422`, becomes a branch on PostHog's error name `validation_error`, never on a status code: PostHog's statuses for invalid sends differ from the provider's, and `recipient_suppressed` is a `validation_error` with status `422`. Handle `recipient_suppressed` before that branch, as the provider's `mapping.md` says. | ||
| - A branch on the provider's rate limit, such as a retry, a wait, or a re-queue on status `429`, becomes a branch on PostHog's status code `429` with a code other than `quota_limited`. PostHog answers a used-up sending quota with status `429` and the code `quota_limited`, which no retry fixes: it takes the path of every other error. | ||
| - A branch on the provider's rate limit, such as a retry, a wait, or a re-queue on status `429`, goes, as the bullet on retries says. A `429` that is left after the SDK's retries takes the path of every other error, as `quota_limited` does. |
There was a problem hiding this comment.
Step 7 intentionally sends a final 429 through the generic error path, but the Resend mapping still tells the agent to keep a separate 429 check in Python, Ruby, and Go. Update those examples to follow Step 7 so the agent gets one instruction.
…ppings Step 7 removes a branch on the provider's rate limit, and a 429 left after the SDK's retries takes the path of every other error. The Resend Python, Ruby and Go examples and the SendGrid status-branch rule still kept a 429 check, and the .NET and Elixir name examples used the rate limit. Align them with Step 7 so the agent gets one instruction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@PostHog review |
|
@Silthus PostHog Review is reviewing this pull request. |
…nt retried SendGrid's RetryDelegatingHandler throws only for a status in RetriableServerErrorStatusCodes. A client that left 500 out of that list got a 500 back without a throw, so its moved site must not throw for it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ReliabilitySettings retries 500, 502, 503 and 504 unless its options set others (read from SendGrid 9.29.3 at run time). Without the list, a trial agent guessed 503 and 504 and let a 500 or 502 through. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@PostHog review |
|
@Silthus PostHog Review is reviewing this pull request. |
gewenyu99
left a comment
There was a problem hiding this comment.
🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.
| - **Python.** PostHog's SDK raises too. When the same `try` also holds a SendGrid call that stays, such as a suppression call, keep each `except HTTPError` handler for that call's errors, and add the PostHog exception's handling next to it. | ||
| - **Ruby, PHP, Go, and .NET.** The code read SendGrid's status code. Read PostHog's error the way the rewrite file says under "The answer", and keep each branch: a status of 400 or more becomes "there is an error", a branch on `400` reads PostHog's error name `validation_error`, and a branch on another status, such as 429, reads PostHog's status code. Their SendGrid clients don't raise on a refused send: a site that never read the status went on after one, and only a network failure raised (Ruby, PHP, .NET) or set `err` (Go). A .NET client whose options set `HttpErrorAsException` is the exception: it threw on a refused send, so its sites throw PostHog's exception the way they threw before, and the next sentence does not apply to them. Such a site logs PostHog's error and goes on, and raises or returns it when its code is `network_error`, as Step 7 says. A Go site that sent through `MakeRequestRetry` or `MakeRequestRetryWithContext` also got `err` when SendGrid still answered 429 after its last retry (`rate limit retry exceeded`): it also returns PostHog's error when its status code is 429, which PostHog's send answers only once its own retries are used up. A .NET site whose SendGrid client retries server errors stays, and the reason is `its client retries server errors`: the client retried a server error, SendGrid's `RetryDelegatingHandler` threw `HttpRequestException` after its last retry, and PostHog's send does neither. The client retries when its options set `ReliabilitySettings` with `MaximumNumberOfRetries` above 0 and it makes its own `HttpClient`, or when the code adds a handler that retries to the `HttpClient` the client gets, the one it passes to the client or the one of the client `AddSendGrid` registers: `RetryDelegatingHandler` with `MaximumNumberOfRetries` above 0, a Polly policy, or a resilience handler such as `AddStandardResilienceHandler`. `ConfigureHttpClientDefaults` adds its handlers to every client from .NET's factory, and often sits in another project, such as an Aspire app's service defaults: search the whole solution for it before a site whose `HttpClient` comes from .NET's factory moves, an `AddSendGrid` site or one given `CreateClient`'s client alike. `AddSendGrid` gives the client an `HttpClient` from .NET's factory, so the `ReliabilitySettings` in its options retry nothing by themselves. When the code can't tell what a handler it adds there does, or how many times it retries, the site stays too. | ||
| - PostHog refuses a send to a suppressed address with `recipient_suppressed`, where SendGrid took it and dropped it later. When the site throws, retries, or reports a failure on any error, handle `recipient_suppressed` before that branch the way the site handled a send that went out: log it and go on. Note each site that got this branch in the report. | ||
| - **Ruby, PHP, Go, and .NET.** The code read SendGrid's status code. Read PostHog's error the way the rewrite file says under "The answer", and keep each branch but one on `429`, which Step 7 removes: a status of 400 or more becomes "there is an error", a branch on `400` reads PostHog's error name `validation_error`, and a branch on another status reads PostHog's status code. Their SendGrid clients don't raise on a refused send: a site that never read the status went on after one, and only a network failure raised (Ruby, PHP, .NET) or set `err` (Go). A .NET client whose options set `HttpErrorAsException` is the exception: it threw on a refused send, so its sites throw PostHog's exception the way they threw before, and the next sentence does not apply to them. Such a site logs PostHog's error and goes on, and raises or returns it when its code is `network_error`, as Step 7 says. A Go site that sent through `MakeRequestRetry` or `MakeRequestRetryWithContext` also got `err` when SendGrid still answered 429 after its last retry (`rate limit retry exceeded`): it also returns PostHog's error when its status code is 429, which PostHog's send answers only once its own retries are used up. A .NET site whose SendGrid client retries moves too: remove the retry setup that only that client used, `ReliabilitySettings` in its options or a retry handler on its `HttpClient`, and never touch `ConfigureHttpClientDefaults` or a handler that other clients share. A client that `ReliabilitySettings` set up threw once its last retry failed: SendGrid's `RetryDelegatingHandler` throws `HttpRequestException` for a status in its `RetriableServerErrorStatusCodes` that is left, `500`, `502`, `503`, and `504` unless the options set others, so its site throws PostHog's error for those statuses. |
There was a problem hiding this comment.
I think moved .NET sites that got their SendGrid client from services.AddSendGrid(...) will newly throw on a server error, and standalone sites will lose their catch (TimeoutException), because:
AddSendGridhands anIHttpClientFactoryHttpClientto SendGrid'sBaseClient, which runshttpClient ?? CreateHttpClientWithRetryHandler(options).ReliabilitySettingsnever ran for those clients, so their sites never threw on a5xx, and this rule still makes them throw.RetryDelegatingHandleralso passes the request straight through whenMaximumNumberOfRetriesis0.- Where the handler did run, it caught
TaskCanceledExceptionafter its last try and threwSystem.TimeoutException.PostHogClientletsOperationCanceledExceptionthrough, so acatch (TimeoutException)around the send stops running.
Scope the throw rule to a client built without an injected HttpClient and with MaximumNumberOfRetries above 0, and map a catch (TimeoutException) on those sites to OperationCanceledException.
There was a problem hiding this comment.
This mapping file should honestly not be a markdown 👁️ Maybe a json/yaml what ever
| - When the old code caught only some errors, such as one error class or one status, catch PostHog's error and branch on its `code` or status code for those, and raise or return every other error as before. Several handlers of the provider's error classes on one `try` become one catch of PostHog's error that branches in their order: the first of several catches of the same error type would take every error. Never widen a catch. PostHog reports a network error with the same error type as an API error, with the code `network_error`. When the old catch took only the provider's answers, such as an HTTP error class, and a network failure passed it, raise or return `network_error` the way the old code let a network failure through. When a later handler of the same `try`, such as a broader `rescue` or `except` after it, took that network failure, run that handler's code for `network_error`: a raise inside one handler skips the handlers after it. A catch of every error around the send, such as a bare `except:`, `except Exception`, `rescue => e`, or `catch (\Throwable $e)`, caught network failures before too: it stays as it is, and lets no `network_error` through, here and in the bullet above. | ||
| - When the old code told one kind of network failure from another, the site stays: `moves: no — catches one kind of network failure`. That is a branch on the transport's error code or class, such as `err.code === 'ECONNRESET'` or `'ETIMEDOUT'` in Node.js, `requests.Timeout`, `requests.ConnectionError`, or `e.errno` in Python, `Faraday::ConnectionFailed`, `Net::ReadTimeout`, or `Errno::ECONNRESET` in Ruby, or curl's error number in PHP. Every SDK retries network failures and reports what is left as one `network_error`, which doesn't say which kind it was, so the branch would never run. A catch that takes every network failure alike, such as a `catch` around `fetch`, which throws only when no answer arrived, is not such a branch: it becomes a branch on `network_error`, as above. The one exception is the timeout of a deadline the site keeps, where the send takes the deadline itself: | ||
| - When the old code told one kind of network failure from another, the site stays: `moves: no — catches one kind of network failure`. That is a branch on the transport's error code or class, such as `err.code === 'ECONNRESET'` or `'ETIMEDOUT'` in Node.js, `requests.Timeout`, `requests.ConnectionError`, or `e.errno` in Python, `Faraday::ConnectionFailed`, `Net::ReadTimeout`, or `Errno::ECONNRESET` in Ruby, or curl's error number in PHP. Every SDK retries network failures and reports what is left as one `network_error`, which doesn't say which kind it was, so the branch would never run. A catch that takes every network failure alike, such as a `catch` around `fetch`, which throws only when no answer arrived, is not such a branch: it becomes a branch on `network_error`, as above. The one exception is the timeout of a deadline the site keeps, where the send takes the deadline itself, never one the code set for each try: | ||
| - Go: `Send` ends at the context's deadline with a `*posthog.EmailError` that unwraps to the request's error, so `errors.Is(err, context.DeadlineExceeded)`, and a `net.Error` read with `errors.As` whose `Timeout()` is true, still match. |
There was a problem hiding this comment.
I think retrying Go sites that check errors.As(err, &netErr) && netErr.Timeout() will miss a context deadline that fires during the wait before a retry, because:
- A Go deadline send that keeps the SDK's retries runs
posthog-go's backoff under the caller'scontext.WithTimeout. - When the first try fails with a
net.Errorthat is no timeout, such asECONNREFUSED, and the context ends during the wait,posthog-gojoins them witherrors.Join(e.cause, context.DeadlineExceeded). errors.As(err, &netErr)finds the first try's*url.Errorfirst, sonetErr.Timeout()isfalse.
| - Go: `Send` ends at the context's deadline with a `*posthog.EmailError` that unwraps to the request's error, so `errors.Is(err, context.DeadlineExceeded)`, and a `net.Error` read with `errors.As` whose `Timeout()` is true, still match. | |
| - Go: `Send` ends at the context's deadline with a `*posthog.EmailError` that unwraps to `context.DeadlineExceeded`, so `errors.Is(err, context.DeadlineExceeded)` still matches. When `Send` keeps retries and the deadline ends a retry wait after a non-timeout `net.Error`, `errors.Join` puts the first `net.Error` ahead of `context.DeadlineExceeded`, so rewrite a `net.Error` `Timeout()` check to also match `errors.Is(err, context.DeadlineExceeded)`. |
| - On a self-hosted instance, also set the app host, under the name the rewrite file gives, to the instance's app URL. The email send goes only to the app host, and without it the send fails with `missing_app_host`. | ||
| - Put the email client in its own server-only module, next to the provider's client, and use it where the sites send. | ||
| - A site whose request had its own deadline keeps it, as the provider's `mapping.md` says under "HTTP": the send gets that deadline, and no retries, since the request never retried. Every SDK otherwise waits up to 15 seconds for each request and retries twice, which can run past the site's deadline. The rewrite file names the timeout and retry options under "A deadline". Where the send takes the deadline itself, as Go's context and .NET's cancellation token do, pass it there; the SDK still ends each request at its own timeout, so a deadline longer than that also needs a client of its own with the timeout set to it. Otherwise each distinct deadline gets an email client of its own with that timeout, in the same module, and its sites send through it. Sites without a deadline keep the first client. Never give a site a deadline it didn't have, or change one it had. A request with its own deadline whose HTTP client the code also set up to retry a POST did retry, and the send that keeps the deadline takes no retries: the site stays, `moves: no — the PostHog SDK can't keep the request's retries`. Such setups include `axios-retry`, which by default retries a network error on any method, a `requests` session adapter with a urllib3 `Retry` or a number as its `max_retries`, such as `HTTPAdapter(max_retries=3)`, which by default retries a failed connection on any method, or Faraday's `retry` middleware with `:post` in its `methods:` or a `retry_if`. Read the options the code gives the setup, such as `axios-retry`'s `retries` and `retryCondition`, or `Retry`'s `total`, `connect` and `allowed_methods`, where a number stands for `total`, to tell whether it retries the POST. A retry setup that skips a POST, such as Faraday's `retry` with its default methods or a setup with no retries left, never retried the send, and the site keeps its deadline with no retries. Only a limit the code sets is the site's own deadline: a library's default, such as `httpx`'s 5 seconds or `ky`'s 10, is not one, like a provider SDK's default, and such a site sends with the PostHog SDK's own timeout and retries. The SDK keeps a deadline only of the kind its timeout is. Every SDK's timeout limits the whole request, except Python's and Ruby's, which limit each wait on the network, as their rewrite files say. A limit of another kind is one the SDK can't keep: axios's `timeout` in Node.js ends a request only after that long without data, and `ky`'s `timeout` ends once the answer's headers arrive. Every SDK has one timeout for connecting and reading alike, so a request that limits only connecting, or limits connecting and reading apart, such as Faraday's `open_timeout`, Guzzle's `connect_timeout`, `requests`' `timeout=(1, 10)`, or an `httpx.Timeout` with its own `connect`, has a deadline the SDK can't keep, unless the rewrite file says under "A deadline" how its client keeps such a limit, as Go's does. When the SDK can't hold the deadline, as there or with a deadline that is not whole seconds in Rust, the site stays: `moves: no — the PostHog SDK can't keep the request's deadline`. | ||
| - A site whose request had its own deadline keeps it, as the provider's `mapping.md` says under "HTTP": the send gets that deadline, and no retries when the code never retried the request. Every SDK otherwise waits up to 15 seconds for each request and retries twice, which can run past the site's deadline. The rewrite file names the timeout and retry options under "A deadline". Where the send takes the deadline itself, as Go's context and .NET's cancellation token do, pass it there; the SDK still ends each request at its own timeout, so a deadline longer than that also needs a client of its own with the timeout set to it. Otherwise each distinct deadline gets an email client of its own with that timeout, in the same module, and its sites send through it. Sites without a deadline keep the first client. Never give a site a deadline it didn't have, or change one it had. When the code retried the request, around the send or through its HTTP client, such as `axios-retry`, a urllib3 `Retry` adapter, or Faraday's `retry` with `:post`, the send keeps the SDK's own retries, with no zero retries from the rewrite file's "A deadline", and that retry setup goes when nothing else uses it. A deadline the code set for each try is the email client's timeout, never one passed to the send. A deadline over all the tries, such as one `AbortSignal` across them, p-retry's `maxRetryTime`, or tenacity's `stop_after_delay`, is never the client's timeout: the SDK keeps it only where the send takes it, as Go's context and .NET's token do. Only a limit the code sets is the site's own deadline: a library's default, such as `httpx`'s 5 seconds or `ky`'s 10, is not one, like a provider SDK's default, and such a site sends with the PostHog SDK's own timeout and retries. The SDK keeps a deadline only of the kind its timeout is. Every SDK's timeout limits the whole request, except Python's and Ruby's, which limit each wait on the network, as their rewrite files say. A limit of another kind is one the SDK can't keep: axios's `timeout` in Node.js ends a request only after that long without data, and `ky`'s `timeout` ends once the answer's headers arrive. Every SDK has one timeout for connecting and reading alike, so a request that limits only connecting, or limits connecting and reading apart, such as Faraday's `open_timeout`, Guzzle's `connect_timeout`, `requests`' `timeout=(1, 10)`, or an `httpx.Timeout` with its own `connect`, has a deadline the SDK can't keep, unless the rewrite file says under "A deadline" how its client keeps such a limit, as Go's does. When the SDK can't hold the deadline, as there or with a deadline that is not whole seconds in Rust, the site stays: `moves: no — the PostHog SDK can't keep the request's deadline`. |
There was a problem hiding this comment.
I think an agent that follows "A deadline" in the language rewrite guides, or that trusts any HTTP retry adapter it sees, will set the wrong timeouts and retries, because:
- The Go and .NET guides still say to pass the deadline's context or token and set
MaxRetriesto0for every deadline site. This rule keeps the SDK's retries when the code retried, and puts a per-try deadline on the client's timeout. - urllib3's
RetryleavesPOSTout ofallowed_methodsby default, andaxios-retry's default condition skipsPOSTon a5xx. A site with such an adapter never retried its send, yet this rule gives it three SDK tries on a single-try deadline.
Bring the Go and .NET "A deadline" sections in line with this rule, and keep the check that the HTTP client's retry setup really retried POST before the send keeps the SDK's retries.
| - An error's name is a coarse category with Resend's error names: `validation_error`, `missing_api_key`, `invalid_permission`, `not_found`, `concurrent_idempotent_requests`, `invalid_idempotent_request`, `rate_limit_exceeded`, `monthly_quota_exceeded`, `service_unavailable`, or `application_error`. Its code is PostHog's own, such as `recipient_suppressed`, `sender_not_verified`, or `quota_limited`: branch on the code. A send to a suppressed address fails with the code `recipient_suppressed` and the name `validation_error`. | ||
| - The answer carries no response headers. An error carries the retry delay PostHog named, when it named one, under the name the rewrite file gives under "The answer". | ||
| - Every SDK retries network errors and rate limits by itself, with the same idempotency key. A network error that is left after the retries comes back as the same error type as an API error, with the code `network_error`. | ||
| - PostHog owns delivery. A send that succeeds is queued, not delivered: PostHog retries the delivery itself, and the answer says nothing about it. Every SDK retries, with the same idempotency key, only the answers that mean PostHog didn't take the send: a network error, a rate limit (`429` with a code other than `quota_limited`), and `503`. Every other error is final. A network error that is left after the retries comes back as the same error type as an API error, with the code `network_error`. |
There was a problem hiding this comment.
I think HTTP-boundary tests that Step 8 rewrites for a 409 idempotency_key_in_flight will fail on a retry request they don't expect, because:
- The SDKs also retry
409when the code isidempotency_key_in_flight, and don't retry429upload_quota_exceeded. I checkedposthog-go'semails_error.go, and haven't checked the other seven. - Step 8 uses this list to decide which mocked statuses need three stubbed answers and which need one.
| - PostHog owns delivery. A send that succeeds is queued, not delivered: PostHog retries the delivery itself, and the answer says nothing about it. Every SDK retries, with the same idempotency key, only the answers that mean PostHog didn't take the send: a network error, a rate limit (`429` with a code other than `quota_limited`), and `503`. Every other error is final. A network error that is left after the retries comes back as the same error type as an API error, with the code `network_error`. | |
| - PostHog owns delivery. A send that succeeds is queued, not delivered: PostHog retries the delivery itself, and the answer says nothing about it. Every SDK retries, with the same idempotency key, only the answers that mean PostHog didn't take the send: a network error, an idempotency key still in flight (`409` with the code `idempotency_key_in_flight`), a rate limit (`429` with a code other than `quota_limited` or `upload_quota_exceeded`), and `503`. Every other error is final. A network error that is left after the retries comes back as the same error type as an API error, with the code `network_error`. |
… list to what the SDKs do - SendGrid .NET: only a client that built its own HttpClient with retries above 0 threw; AddSendGrid and an injected HttpClient never ran ReliabilitySettings. Its catch (TimeoutException) becomes catch (OperationCanceledException). - Go: a deadline that ends the wait before a retry joins the last try's net.Error ahead of context.DeadlineExceeded, so a Timeout() check also checks errors.Is. - Retried answers name 409 idempotency_key_in_flight, which all eight SDKs retry. - Restore the check that an HTTP client's retry setup covers the POST. - Go and .NET "A deadline": zero retries only when the code never retried. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s on the context, unknown outcome of a network error - sendgrid/mapping.md: drop the Go MakeRequestRetry sentence that returned PostHog's error on any 429; a 429 left after the SDK's retries takes the path of every other error, as Step 7 says. - rewrite-go.md and rewrite-dotnet.md: the context or token carries a deadline over the whole send; a deadline set for each try is EmailRequestTimeout. - 7-rewrite.md: a network error leaves open whether PostHog took the send. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s answer, no Resend rate-limit branch, .NET fallback handlers stay Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the SDK's retries stays A deadline can end the wait before a retry, so PostHog's error can't tell the last try's timeout from the deadline. Keep main's rule for a send with no retries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…before moving it A handler registered elsewhere, in AddHttpClient or ConfigureHttpClientDefaults, can answer for a failed send. When its behaviour can't be read, the site stays on SendGrid (decision d-2deba2ad). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er chain is traced A factory or handler from a referenced assembly can install a fallback the solution doesn't show, so the site stays and the reason names where tracing stopped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A 429 branch that does more than retry the send keeps that work. - A RetryDelegatingHandler added by hand to an injected HttpClient threw like ReliabilitySettings, so its site throws the same way. - A 503 can come back after PostHog queued the send (Django's CDP call timing out after the insert, or async intake without its ack), so a site that falls back to another provider on a failed send stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…429 branch, .NET timeout catch - Step 3: a branch whose failed send falls back to another provider stays, so the stay rule applies before Step 7. A 409 still in flight leaves the outcome open too. - Step 7: a 429 branch that does more than retry keeps only that work, on a 429 other than quota_limited, then goes on as it did. - SendGrid .NET: catch (TimeoutException) becomes catch (OperationCanceledException) only when the site keeps a deadline over the whole send; otherwise it caught the client's own timeout, which PostHog reports as network_error, so the site stays. SendGrid's own RetryDelegatingHandler is no unreadable referenced-assembly handler. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…9 branch stays A branch that only gives its caller a rate-limit answer goes with the retry, as before. Work besides it, such as recording a throttle event, stays on a 429 other than quota_limited and then takes the ordinary error path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… stays - Step 7: a retry that can wait more than 30 seconds stays, since every SDK gives up at once on a Retry-After over 30 seconds; so does a retry that does work on each failed try, which the SDK's retries can't run. - Resend mapping: a rate_limit_exceeded branch goes, except work besides its retry, as Step 7 says. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts before moving - Step 7: decide whether the retry rule keeps a site, by a wait past 30 seconds or work on each failed try, before the 429 bullet; a resend left to a scheduler counts, a log line is no per-try work. - SendGrid .NET: catch (TimeoutException) becomes catch (OperationCanceledException) only when the solution shows the site's deadline ends before the HttpClient's own Timeout. A hand-added RetryDelegatingHandler throws only with retries above 0; set to 0, or to a count you can't read, it keeps the site on SendGrid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
When
migrate-emailmoved a send, it kept the app's own retry around it: a retry loop, a retry library such as p-retry or tenacity, or a wait and resend on429. It even turned a rate-limit retry into a branch on PostHog's429. The PostHog SDKs already retry, with the same idempotency key, the answers a retry can fix, so the app's retry runs on top of the SDK's. Two kinds of sites stayed on the provider only because their client retried: a .NET SendGrid client withReliabilitySettingsor a retry handler, and a site with its own deadline whose HTTP client retried the POST. In migration trials, agents also kept adding their own retries around the send.Caution
🔴 Merge risk: high, graded at full use, since the skill sets how apps send on the critical email path · two-way door until the next context-mill release ships the skill; one-way once agents run it, since the rewrite stays in each app
429, or a re-queue done only to retry: on the base, the agent kept the retry around PostHog's send and turned a rate-limit retry into a branch on PostHog's429. At HEAD the agent removes it and adds none, and what the code did after its last retry failed runs on PostHog's error. Run at HEAD only: the trial on generated codebases (below) runs each moved send against a fake PostHog, and every answer takes only the SDK's tries. That each caller fails the way the old code did after its last retry is read from the old code, not run on it, so this row counts as uncovered: an agent that misreads the old failure path changes how that site fails. A BullMQ worker's re-queue on429goes while the job's ownattemptsstays, so a429that is left reruns the job. Uncovered: a re-queue that does more than retry the send, which the rule keeps. An agent that misreads one as retry-only would silently drop that work. No data bounds how many apps have one, so this row sets the grade.Retry-Afterover 30 seconds (checked in all eight at origin HEAD), so moving it would lose that recovery. At HEAD the site stays,moves: no — waits longer than the SDK retries, and so does a retry that writes attempt state or runs recovery on each failed try,moves: no — does work on each failed try(d-d3f3bd05). A retry with more tries than the SDK's two, all within 30 seconds, still moves and now makes fewer tries, so a send that only its third or later try got through now fails (Michael's 2026-11-17 decision); read from the text, not run. The agent decides the stay before the429rule, whoever runs the resend, so a branch that returnsFalsefor a scheduler to send again an hour later stays. Read from the text, not run.ReliabilitySettingsor a retry handler on itsHttpClient: on the base it stayed on SendGrid. At HEAD it moves, and the retry setup that only that client used goes.ConfigureHttpClientDefaultsand handlers other clients share are never touched. A client thatReliabilitySettingsset up, with retries above0and noHttpClientpassed in, threw once its last retry failed, so its site throws PostHog's error for a status in that client'sRetriableServerErrorStatusCodes,500,502,503and504unless its options set others. Itscatch (TimeoutException)becomescatch (OperationCanceledException)only when the solution shows the site's deadline over the whole send ends before theHttpClient's ownTimeout; otherwise it could have caught the client's own timeout, which PostHog reports asnetwork_errorlike any other network failure, so the site stays (posthog-dotnet17b0d4c,EmailSender.cs:597). Such a site that recovered from a transient500,502or504through its old retries now fails on the first one, since the PostHog SDKs retry only503of these (Known issues, Michael's 2026-11-17 decision). ARetryDelegatingHandleradded by hand to an injectedHttpClientwith retries above0threw the same way, so its site throws the same way; one set to0, or to a count the agent can't read, keeps the site on SendGrid. A client fromservices.AddSendGrid, or one built with its ownHttpClient, never ran those settings, so its site logs a refused send and goes on, as before. Run at HEAD only, in the trial's .NET codebases compiled against a fake PostHog: the standalone client's site throws for a 500 and a 503 and logs a 400 or a 429; theAddSendGridsite logs every status; the site that caughtTimeoutExceptioncaught the 5-second token's cancellation at4bed508. Sincec400052that site stays on SendGrid, read from the text, not re-run: SendGrid builds itsHttpClient, so the solution can't show that the token ends before the client's own timeout. That each matches its old client is checked against SendGrid's source, not run on the base. Uncovered by a paired run. A handler that does more than retry, such as a Polly fallback that answers for a send its retries failed, keeps the site on SendGrid, and the reason names it: the trial's Polly-plus-fallback site stays with its code unchanged. Uncovered: a handler whose behaviour the agent can't read, such as one registered elsewhere in the solution; an agent that misses or misreads it moves a site whose failures that handler answered.axios-retryor a urllib3Retryadapter, which retry a network error or a failed connection on any method by default, or Faraday'sretrywith:post: on the base it stayed. A setup whose options skip the POST never retried the send, and that site keeps its deadline with no retries, as on the base. At HEAD it moves with its deadline and the SDK's own retries, with no zero retries from the rewrite file, and the retry setup goes when nothing else uses it. A deadline the code set for each try becomes the email client's timeout, so its timeout is anetwork_error, in .NET too: a site that catches only the timeout stays, as it does for any branch on one kind of network failure. A deadline over all the tries, such as oneAbortSignalacross a loop, p-retry'smaxRetryTime, or tenacity'sstop_after_delay, is never the client's timeout: it moves only where the send takes it, Go's context and .NET's token, and elsewhere the site stays, as on the base. Run at HEAD only, so uncovered by a paired run: the trial's Python codebase with a urllib3Retryadapter andtimeout=5: a send that answers after 7 seconds is cut at 5 seconds on each of the SDK's 3 tries. Also at HEAD: a Node.js loop with oneAbortSignal.timeoutacross its tries and a tenacitystop_after_delaysite both stay, and so does a .NET loop with a new token for each try whose caller catches only the timeout. Also at HEAD: a .NET SendGrid client withReliabilitySettingsand one token on the send moves, passes the token to the send with the SDK's retries, and is cancelled at the same 5 seconds. Also at HEAD: a Go loop under one 5-second context with anet.ErrorTimeout()check stays,moves: no — catches one kind of network failure, because a deadline that ends the wait before a retry leaves PostHog's error unable to tell the last try's timeout from the deadline; a Go site with no loop, one 5-second context and the same check moves with zero retries and hits the check at 5 seconds. A Python site whose urllib3Retrykeeps its default methods counts as retrying, since it retried a failed connection. Uncovered by a paired run: the Go no-retry check is read from posthog-go's code and a run of it, not run on the base.429: on the base it became a branch on PostHog's429with a code other thanquota_limited. At HEAD the branch goes, and a429that is left after the SDK's retries takes the path of every other error, asquota_limiteddid (the conductor's order). Work the branch did besides its retry or rate-limit answer, such as recording a throttle event, stays and runs on a429other thanquota_limitedonce the SDK's retries are spent, before that path (d-ffbf97bb). Read from the text, not run. Run at4bed508: a Resend Python site whoseRateLimitErrorbranch logged and returnedFalsefor its scheduler to try again next hour moved without that branch. Since72b4041that site stays on Resend, read from the text, not re-run, because its resend waits past the SDK's 30 seconds. Uncovered: a branch that answers its own caller with a429; that site now answers a rate limit the way it answers any other failed send.503can come back after PostHog queued the send: Django's call to the CDP service can time out after the queue insert, and in async intake the API answers503without the broker's ack (posthog2d02be1,transactional_email.py:740-802,intake-producer.ts:80-107). The SDK's retries reuse the idempotency key, and a replay returns the first result, so they never send twice. A site that falls back to another provider when the send fails could send the same email twice, so it stays from Step 3 on,moves: no — falls back to another provider when the send's outcome is unknown. A409still in flight leaves the outcome open the same way. Read from PostHog's code, not run.mainshows only themigrate-email-resendandmigrate-email-sendgridbundles'7-rewrite.md,9-report.md, providermapping.md,rewrite-go.mdandrewrite-dotnet.md, plusmanifest.json, change.Rollout
409idempotency_key_in_flight, a429other thanquota_limited, and503), and an app's own retry around the send goes (Michael, 2026-11-17). A rate-limit branch on the provider's429goes, and a429left after the SDK's retries takes the path of every other error (the transactional email conductor's order).Changes
7-rewrite.md, "What every SDK's send takes and answers": PostHog owns delivery. A send that succeeds is queued, not delivered. Every SDK retries, with the same idempotency key, only a network error, a409with the codeidempotency_key_in_flight, a429with a code other thanquota_limited, and503; every other error is final. These are what all eight SDKs share; posthog-rs also retries a429upload_quota_exceeded, which a send without attachments never gets.7-rewrite.md, "Rewrite each site": a new rule removes the app's own retry around the send and adds none. What ran after the last retry now runs on PostHog's error. A retry that can wait more than 30 seconds, a resend left to a scheduler included, or that does work on each failed try, keeps its site on the provider; the agent decides this before the429rule. A job framework that reruns a whole job is left alone. The rate-limit branch no longer survives as a branch on PostHog's429: a429left after the SDK's retries takes the path of every other error. Work the branch did besides its retry or rate-limit answer stays, on a429other thanquota_limited.3-inventory.mdand7-rewrite.md, a503or409after queueing: after a network error, a409still in flight, or a503, nobody knows whether PostHog took the send, so a branch whose failed send falls back to another provider stays from Step 3 on.7-rewrite.md, "Set up the client": a site with its own deadline whose code retried the request moves, keeps its deadline and the SDK's retries, and loses its retry setup when nothing else uses it. Go's and .NET's zero retries for a deadline send apply only when the code never retried. A deadline set for each try becomes the email client's timeout; a deadline over all the tries is kept only by a send that takes it, Go's context or .NET's token, and elsewhere the site stays. A per-try deadline's timeout is anetwork_errorin every SDK. The check that a retry setup covers the POST stays, with the defaults ofaxios-retryand urllib3'sRetry. A Go timeout branch keeps both of its checks on a send with no retries, as onmain. A Go site that keeps the SDK's retries and branches on a timeout stays,moves: no — catches one kind of network failure, since a deadline can end the wait before a retry. A network error or a503leaves open whether PostHog took the send, so a site that falls back to another provider on a failed send stays.sendgrid/mapping.md: a .NET site whose SendGrid client retries moves, and the retry setup only that client used goes; the detection passage behind the old stay rule is cut. A client thatReliabilitySettingsset up, with retries above0and noHttpClientpassed in, asservices.AddSendGridpasses one, threw once its last retry failed, so its site throws PostHog's error for a status in that client'sRetriableServerErrorStatusCodes,500,502,503and504unless its options set others, and itscatch (TimeoutException)becomescatch (OperationCanceledException)only when the solution shows the site's deadline ends before theHttpClient's ownTimeout; otherwise the site stays. ARetryDelegatingHandleradded by hand to an injectedHttpClientwith retries above0throws the same way; set to0, or to a count the agent can't read, it keeps the site on SendGrid. A handler that does more than retry, such as a fallback that answers for a send its retries failed, keeps the site on SendGrid, and the reason names the handler. Before moving any .NET site, the agent traces every handler on the client'sHttpClient, fromAddHttpClient,ConfigureHttpClientDefaultsand any factory that builds it, and the site stays unless the solution shows the whole chain and what each handler does; the reason names the handler or the registration where tracing stopped. ARetryDelegatingHandleradded by hand to an injectedHttpClientthrows likeReliabilitySettings. The Node.jsquota_limitedcheck onerr.code === 429and the HTTP paragraph's "no longer retries" note are cut. Ruby, PHP, Go and .NET keep each status branch but one on429, which Step 7 removes. The GoMakeRequestRetrysentence is cut: a429left after the SDK's retries takes the path of every other error.resend/mapping.md: "The retry delay" no longer keeps a deadline site's own wait and retry. Python'sRateLimitError, Ruby'sRateLimitExceededErrorand Go's rate-limit check no longer become a check of status429: Step 7 removes them, and so does a Node.js branch onrate_limit_exceeded, except work besides its retry, which stays. The .NET and Elixir name examples usevalidation_errorinstead of the rate limit.mapping.mdfiles, suppressed recipients: "throws, retries, or reports a failure" drops "retries".rewrite-go.mdandrewrite-dotnet.md, "A deadline": zero retries on a deadline send only when the code never retried the request, as Step 7 says, and the deadline passed to the send is the site's deadline over the whole send, never one the code set for each try.7-rewrite.md"Write down" and9-report.md: each retry the move removed is noted under the sites table and listed in the report, next to the opens-off and click notes frommain.Known issues
context/skills/migrate-email/references/sendgrid/mapping.md:156: a SendGrid .NET site whoseReliabilitySettingsretried a transient500,502or504now fails on the first such answer from PostHog, because every PostHog SDK retries only a network error,409idempotency_key_in_flight, a429other thanquota_limited, and503. This follows Michael's 2026-11-17 decision that the app's own retry goes and the SDK retries what a retry can fix, kept in d-2deba2ad.context/skills/migrate-email/references/sendgrid/mapping.md:156: no trial covers a .NET solution whose handler orHttpClientfactory comes from a referenced assembly. The rule keeps such a site on SendGrid, read from the text; trial H covers only a handler in the solution whose behaviour can't be read. No trial covers aRetryDelegatingHandleradded by hand to an injectedHttpClient. The .NET timeout trial site, withReliabilitySettings, a 5-second token andcatch (TimeoutException), moved at4bed508and now stays on SendGrid, because the solution can't show itsHttpClient's timeout; no run atc400052shows that stay, or tells it from a site whose token provably ends first.context/skills/migrate-email/references/3-inventory.md:27andcontext/skills/migrate-email/references/7-rewrite.md:32: no trial runs a site that falls back to another provider when its send fails. Steps 3 and 7 keep it on the provider; the503after queueing is read from PostHog's code.context/skills/migrate-email/references/7-rewrite.md:127: no trial runs a429branch that does other work besides its retry againstrate_limitedandquota_limited; the rule is read from the text.context/skills/migrate-email/references/7-rewrite.md:118: the stay rules for a retry that waits past 30 seconds or does work on each failed try sit in Step 7, after the client setup, so an agent can install or upgrade the PostHog SDK for a site it then keeps on the provider. The site's send is untouched; only the dependency is extra.c400052as d-2a5e8d62, with no further commits.How it was tested
The lane gate on
c400052,72b4041,8b194e7,ffb093aand4bed508(withmainmerged after feat(migrate-email): map provider tracking settings to tracking and sensitive #486):vitest run scripts, 400 tests,bun run test:skills,bun run test:pluginsandnode scripts/build.jspass, with a clean tree after the build. Both bundles carry the tracking text, the opens-off rule and the opens-first sentence frommain, and this PR's retry text.Shadow diff from
main(ee1402d) to the head,c400052, rebuilt fresh on both sides: only3-inventory.md,7-rewrite.md,9-report.md, the providermapping.md,rewrite-go.mdandrewrite-dotnet.mdof themigrate-email-resendandmigrate-email-sendgridbundles, andmanifest.json, change; every other bundle is byte-identical. Saved in proof storage.Two trial outcomes below change with
72b4041andc400052, read from the text, not re-run: the SendGrid .NET site withReliabilitySettings, one 5-second token andcatch (TimeoutException)now stays on SendGrid, because SendGrid builds itsHttpClientand the solution can't show that client's timeout. The Resend Python site whoseRateLimitErrorbranch returnsFalsefor a scheduler to try again next hour now stays on Resend, because its resend waits past 30 seconds. Both now leave the old send untouched; the other rows hold.Live run at the head,
4bed508: the built bundle ran with headless Claude Code (Opus 5.5, high effort) on every trial site the changes since97cc1fdreach. No retry library, loop, wait or429branch is left around any moved send, and each moved code send also carries the sensitive flag and opens-off tracking frommain:ReliabilitySettings500,502,503,504andnetwork_errorservices.AddSendGridwithReliabilitySettingsReliabilitySettings, one 5-second token, andcatch (TimeoutException)OperationCanceledExceptionnet.ErrorTimeout()checkffb093ait stays on SendGrid,moves: no — catches one kind of network failureRetrywith its default methods andtimeout=5except RateLimitErrorlogs and returnsFalse429429branchAt
ffb093a, after final read 9 and rounds 2 to 4, five more trials ran with the built bundle (Opus 5.5, high), each moved send against a fake PostHog:rate_limit_exceededbranch returning its own result429rate_limitedmakes 3 requests and returns the ordinary failure result, a200succeeds202on itsHttpClientmoves: no — catches one kind of network failureTimeout()branchMaxRetries: posthog.Ptr(0); a 7-second answer hits the branch at 5 s with 1 request, a503takes 1 requestReliabilitySettings, one 5-second token,catch (TimeoutException)OperationCanceledException; a slow send is caught at 5 s, a500throws after 1 request, a503after 3At
94be42b, after Michael's d-2deba2ad, two more trials ran, carried to5e3ce3a, which only widens the .NET stay rule:welcome email failedand returning{ ok: false }after the last503makes 3 requests, logs the line and returns{ ok: false }, a400takes 1 request, a200returns{ ok: true }; the Step 9 report names the removed loopAuditHandleradded byConfigureHttpClientDefaultsthat calls anIAuditSinkwith no implementation in the solutionAuditHandler, no file changedEarlier, at
4bed508, the three .NET sites ran against a fake PostHog: 3 requests for a 503 and a rate-limited 429, 1 for every other answer; the standalone client's sites throw for a 500 and a 503, theAddSendGridsite logs every status, and the token site catches the cancellation at 5 seconds. Saved in proof storage as the live run.Live run at
97cc1fd, for the429and .NET status changes: the same harness on every trial site they reach:except RateLimitErrorlogs and returnsFalseso a scheduler tries again next hour429429branch aroundemails.sendReliabilitySettings500,502,503,504andnetwork_errorReliabilitySettingsand one 5-second token onSendEmailAsyncEach moved site then ran against a fake PostHog: 3 requests for a 503 and a rate-limited 429, 1 for every other answer. The Resend site raises for every error after the SDK's tries; where the old code returned
Falseon a rate limit, it now raises. Both .NET sites throw for a 500 or a 503 and log every other refused send, as their old clients did, and the token cancels at 5 seconds. SendGrid's default list was read from SendGrid 9.29.3 at run time. One real send through the moved Resend site arrived in the test inbox, sent from a verified sender: the fixture's own sender is not verified in the test project, and PostHog refused it. The other sites below are Node.js or Python SendGrid sites, a Resend Node.js site, or sites that stay before any status branch, so97cc1fddoes not reach them.Live run at
4ecadc2: the built bundle ran with headless Claude Code (Opus 5.5, high effort) on every trial site the deadline rules can reach, plus the p-retry site for the real send. The account steps were taken as done, and nobody answered questions. A scan of each codebase's source then looked for a retry library or decorator, a loop, a wait, or a429branch in each file that sends:p-retryaroundsgMail.send429, no deadline429, 5 attemptsRetryadapter andtimeout=5ReliabilitySettingsand one 5-second token onSendEmailAsyncAbortSignal.timeout(5000)across 3 triesstop_after_delay(10)Behaviour against a fake PostHog, at
4ecadc2, for the sites above it: each moved site's own module ran its send once per answer: 200, 400, 422recipient_suppressed, 500, 503, 429rate_limited, 429quota_limited, and a 7-second answer for the deadline sites. Node.js ran on this machine, Python in apython:3.12-slimcontainer, and the .NETReceiptMailer.cscompiled in the .NET 8 SDK image. Every site made 3 requests for a 503 and a rate-limited 429, which are the SDK's own tries, and 1 for every other answer, so no app retry is left. Each caller then threw, raised, or logged and went on exactly as its old code did after its last retry. The slow answer ended the Python site after 3 tries of 5 seconds withnetwork_error, and the .NET site at 5 seconds withTaskCanceledException, as its old token did.Real send, at
4ecadc2: the moved p-retry app ran with a project secret key, and onePOST /signupsent through PostHog's US Cloud. The email arrived in the test inbox in the same second. Saved in proof storage as the live run.Earlier live run, at
393fefe, on eight codebases: sites without a deadline, whose rewrite nothing since has changed, with the same behaviour runs and a delivered real send:p-retryaroundsgMail.send@retryon the sendRetry-Afterand resends on429429, 5 attemptsReliabilitySettingsasync-retryaroundresend.emails.sendrequestssession with a urllib3Retryadapter andtimeout=5429,attempts: 5on the jobattemptsstaysEvery site moved, and each report names the removed retry. The first trial, at
efbd483, gave the .NET site no throw for a server error, which round 1 caught.Agent context
429other thanquota_limited, and503, never500,502or504409idempotency_key_in_flight; posthog-rs alone also retries429upload_quota_exceeded, left outRetryDelegatingHandlerand503items9bf50deand8b194e7c400052: untried stay rules, the setup order, and its429escalationRetry-After, per-try wrapper work and Resendrate_limit_exceededitems72b4041, one last fixes round, no further roundsInvolvement: no grilling rounds, no prototypes. Review round 1 (GPT-6.1 Sol), final reads 1 to 9 and fix rounds 2 to 4 (GPT-6.2 Sol, two lenses, two families), a reviewer's change request, a second reviewer's tool-assisted comments, and ReviewHog: every finding fixed, except one declined with a reason, and a format change left for a follow-up. Final read 9 covers
4ecadc2to4bed508under map #315's standing review rule; round 3's Go blocker, the third on that check, was fixed by cutting scope back tomain's rule, and round 4's only finding was this description. Round 5, the one fixes round Michael chose, found a blocker in the new .NET sentence, fixed in5e3ce3aso an untraceable handler chain keeps the site, and asked for a referenced-assembly trial, listed under Known issues. Round 6, on4d855fb's 429,RetryDelegatingHandlerand503clauses from Michael's d-ffbf97bb, found three blockers and a major, all fixed in9bf50deand8b194e7: the failover stay moves to Step 3, the kept429work gets one path, a .NET timeout catch without a kept deadline stays, and SendGrid's own handler is no unreadable one. Its trial asks are under Known issues. Round 7, the last fixes round Michael chose in d-d3f3bd05, reviews those fixes and72b4041's stay clauses for a retry that waits past 30 seconds or works on each try. It found three blockers, fixed inc400052: a .NET timeout catch moves only when the solution shows the deadline ends before theHttpClient's own timeout, a hand-added SendGrid handler set to0retries keeps its site, and the retry stay rules are decided before the429rule. Its two trial majors are under Known issues, and no further round runs. Proof captured: shadow diff and live run (trials on generated codebases atffb093a,4bed508and before, their moved sends run against a fake PostHog, and one real send).🤖 Generated with Claude Code