Skip to content

fix(migrate-email): leave send retries to PostHog's SDK - #541

Merged
Silthus merged 25 commits into
mainfrom
fix/migrate-email-posthog-owns-retries
Nov 20, 2026
Merged

Silthus merged 25 commits into
mainfrom
fix/migrate-email-posthog-owns-retries

Conversation

@Silthus

@Silthus Silthus commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem

When migrate-email moved a send, it kept the app's own retry around it: a retry loop, a retry library such as p-retry or tenacity, or a wait and resend on 429. It even turned a rate-limit retry into a branch on PostHog's 429. The PostHog SDKs already retry, with the same idempotency key, the answers a retry can fix, so the app's retry runs on top of the SDK's. Two kinds of sites stayed on the provider only because their client retried: a .NET SendGrid client with ReliabilitySettings or a retry handler, and a site with its own deadline whose HTTP client retried the POST. In migration trials, agents also kept adding their own retries around the send.

Caution

🔴 Merge risk: high, graded at full use, since the skill sets how apps send on the critical email path · two-way door until the next context-mill release ships the skill; one-way once agents run it, since the rewrite stays in each app

  • A moving site with the app's own retry around the send, such as a retry loop, p-retry, async-retry, tenacity, backoff, retriable, Polly, a wait and resend on 429, or a re-queue done only to retry: on the base, the agent kept the retry around PostHog's send and turned a rate-limit retry into a branch on PostHog's 429. At HEAD the agent removes it and adds none, and what the code did after its last retry failed runs on PostHog's error. Run at HEAD only: the trial on generated codebases (below) runs each moved send against a fake PostHog, and every answer takes only the SDK's tries. That each caller fails the way the old code did after its last retry is read from the old code, not run on it, so this row counts as uncovered: an agent that misreads the old failure path changes how that site fails. A BullMQ worker's re-queue on 429 goes while the job's own attempts stays, so a 429 that is left reruns the job. Uncovered: a re-queue that does more than retry the send, which the rule keeps. An agent that misreads one as retry-only would silently drop that work. No data bounds how many apps have one, so this row sets the grade.
  • A retry that can wait longer than 30 seconds before it sends again, such as one that waits out a long rate-limit window: every PostHog SDK gives up at once on a Retry-After over 30 seconds (checked in all eight at origin HEAD), so moving it would lose that recovery. At HEAD the site stays, moves: no — waits longer than the SDK retries, and so does a retry that writes attempt state or runs recovery on each failed try, moves: no — does work on each failed try (d-d3f3bd05). A retry with more tries than the SDK's two, all within 30 seconds, still moves and now makes fewer tries, so a send that only its third or later try got through now fails (Michael's 2026-11-17 decision); read from the text, not run. The agent decides the stay before the 429 rule, whoever runs the resend, so a branch that returns False for a scheduler to send again an hour later stays. Read from the text, not run.
  • A .NET SendGrid site whose client retried, through ReliabilitySettings or a retry handler on its HttpClient: on the base it stayed on SendGrid. At HEAD it moves, and the retry setup that only that client used goes. ConfigureHttpClientDefaults and handlers other clients share are never touched. A client that ReliabilitySettings set up, with retries above 0 and no HttpClient passed in, threw once its last retry failed, so its site throws PostHog's error for a status in that client's RetriableServerErrorStatusCodes, 500, 502, 503 and 504 unless its options set others. Its catch (TimeoutException) becomes catch (OperationCanceledException) only when the solution shows the site's deadline over the whole send ends before the HttpClient's own Timeout; otherwise it could have caught the client's own timeout, which PostHog reports as network_error like any other network failure, so the site stays (posthog-dotnet 17b0d4c, EmailSender.cs:597). Such a site that recovered from a transient 500, 502 or 504 through its old retries now fails on the first one, since the PostHog SDKs retry only 503 of these (Known issues, Michael's 2026-11-17 decision). A RetryDelegatingHandler added by hand to an injected HttpClient with retries above 0 threw the same way, so its site throws the same way; one set to 0, or to a count the agent can't read, keeps the site on SendGrid. A client from services.AddSendGrid, or one built with its own HttpClient, never ran those settings, so its site logs a refused send and goes on, as before. Run at HEAD only, in the trial's .NET codebases compiled against a fake PostHog: the standalone client's site throws for a 500 and a 503 and logs a 400 or a 429; the AddSendGrid site logs every status; the site that caught TimeoutException caught the 5-second token's cancellation at 4bed508. Since c400052 that site stays on SendGrid, read from the text, not re-run: SendGrid builds its HttpClient, so the solution can't show that the token ends before the client's own timeout. That each matches its old client is checked against SendGrid's source, not run on the base. Uncovered by a paired run. A handler that does more than retry, such as a Polly fallback that answers for a send its retries failed, keeps the site on SendGrid, and the reason names it: the trial's Polly-plus-fallback site stays with its code unchanged. Uncovered: a handler whose behaviour the agent can't read, such as one registered elsewhere in the solution; an agent that misses or misreads it moves a site whose failures that handler answered.
  • A site with its own deadline whose HTTP client retried the POST, such as axios-retry or a urllib3 Retry adapter, which retry a network error or a failed connection on any method by default, or Faraday's retry with :post: on the base it stayed. A setup whose options skip the POST never retried the send, and that site keeps its deadline with no retries, as on the base. At HEAD it moves with its deadline and the SDK's own retries, with no zero retries from the rewrite file, and the retry setup goes when nothing else uses it. A deadline the code set for each try becomes the email client's timeout, so its timeout is a network_error, in .NET too: a site that catches only the timeout stays, as it does for any branch on one kind of network failure. A deadline over all the tries, such as one AbortSignal across a loop, p-retry's maxRetryTime, or tenacity's stop_after_delay, is never the client's timeout: it moves only where the send takes it, Go's context and .NET's token, and elsewhere the site stays, as on the base. Run at HEAD only, so uncovered by a paired run: the trial's Python codebase with a urllib3 Retry adapter and timeout=5: a send that answers after 7 seconds is cut at 5 seconds on each of the SDK's 3 tries. Also at HEAD: a Node.js loop with one AbortSignal.timeout across its tries and a tenacity stop_after_delay site both stay, and so does a .NET loop with a new token for each try whose caller catches only the timeout. Also at HEAD: a .NET SendGrid client with ReliabilitySettings and one token on the send moves, passes the token to the send with the SDK's retries, and is cancelled at the same 5 seconds. Also at HEAD: a Go loop under one 5-second context with a net.Error Timeout() check stays, moves: no — catches one kind of network failure, because a deadline that ends the wait before a retry leaves PostHog's error unable to tell the last try's timeout from the deadline; a Go site with no loop, one 5-second context and the same check moves with zero retries and hits the check at 5 seconds. A Python site whose urllib3 Retry keeps its default methods counts as retrying, since it retried a failed connection. Uncovered by a paired run: the Go no-retry check is read from posthog-go's code and a run of it, not run on the base.
  • A branch on the provider's rate limit that does something other than retry, such as answering its own caller with a 429: on the base it became a branch on PostHog's 429 with a code other than quota_limited. At HEAD the branch goes, and a 429 that is left after the SDK's retries takes the path of every other error, as quota_limited did (the conductor's order). Work the branch did besides its retry or rate-limit answer, such as recording a throttle event, stays and runs on a 429 other than quota_limited once the SDK's retries are spent, before that path (d-ffbf97bb). Read from the text, not run. Run at 4bed508: a Resend Python site whose RateLimitError branch logged and returned False for its scheduler to try again next hour moved without that branch. Since 72b4041 that site stays on Resend, read from the text, not re-run, because its resend waits past the SDK's 30 seconds. Uncovered: a branch that answers its own caller with a 429; that site now answers a rate limit the way it answers any other failed send.
  • A 503 can come back after PostHog queued the send: Django's call to the CDP service can time out after the queue insert, and in async intake the API answers 503 without the broker's ack (posthog 2d02be1, transactional_email.py:740-802, intake-producer.ts:80-107). The SDK's retries reuse the idempotency key, and a replay returns the first result, so they never send twice. A site that falls back to another provider when the send fails could send the same email twice, so it stays from Step 3 on, moves: no — falls back to another provider when the send's outcome is unknown. A 409 still in flight leaves the outcome open the same way. Read from PostHog's code, not run.
  • A job framework that reruns a whole job on any error is left alone, as on the base: a rerun sends again unless the site sets an idempotency key, as on the provider.
  • Every other skill is unchanged: the shadow diff from main shows only the migrate-email-resend and migrate-email-sendgrid bundles' 7-rewrite.md, 9-report.md, provider mapping.md, rewrite-go.md and rewrite-dotnet.md, plus manifest.json, change.
  • Tier: critical: the migrate-email skill changes how apps send transactional email, the critical path of email delivery.
  • Switch-off: none: text-only skill change, live for agents once released; fix forward.

Rollout

  • Stages: ships to agents with the next context-mill release; nothing to stage in the skill itself.
  • Decided: the skill names the answers the SDKs retry (a network error, a 409 idempotency_key_in_flight, a 429 other than quota_limited, and 503), and an app's own retry around the send goes (Michael, 2026-11-17). A rate-limit branch on the provider's 429 goes, and a 429 left after the SDK's retries takes the path of every other error (the transactional email conductor's order).
  • Watch: no alert covers what an agent writes into an app; the migrate-email trial on generated codebases is the check.

Changes

  • 7-rewrite.md, "What every SDK's send takes and answers": PostHog owns delivery. A send that succeeds is queued, not delivered. Every SDK retries, with the same idempotency key, only a network error, a 409 with the code idempotency_key_in_flight, a 429 with a code other than quota_limited, and 503; every other error is final. These are what all eight SDKs share; posthog-rs also retries a 429 upload_quota_exceeded, which a send without attachments never gets.
  • 7-rewrite.md, "Rewrite each site": a new rule removes the app's own retry around the send and adds none. What ran after the last retry now runs on PostHog's error. A retry that can wait more than 30 seconds, a resend left to a scheduler included, or that does work on each failed try, keeps its site on the provider; the agent decides this before the 429 rule. A job framework that reruns a whole job is left alone. The rate-limit branch no longer survives as a branch on PostHog's 429: a 429 left after the SDK's retries takes the path of every other error. Work the branch did besides its retry or rate-limit answer stays, on a 429 other than quota_limited.
  • 3-inventory.md and 7-rewrite.md, a 503 or 409 after queueing: after a network error, a 409 still in flight, or a 503, nobody knows whether PostHog took the send, so a branch whose failed send falls back to another provider stays from Step 3 on.
  • 7-rewrite.md, "Set up the client": a site with its own deadline whose code retried the request moves, keeps its deadline and the SDK's retries, and loses its retry setup when nothing else uses it. Go's and .NET's zero retries for a deadline send apply only when the code never retried. A deadline set for each try becomes the email client's timeout; a deadline over all the tries is kept only by a send that takes it, Go's context or .NET's token, and elsewhere the site stays. A per-try deadline's timeout is a network_error in every SDK. The check that a retry setup covers the POST stays, with the defaults of axios-retry and urllib3's Retry. A Go timeout branch keeps both of its checks on a send with no retries, as on main. A Go site that keeps the SDK's retries and branches on a timeout stays, moves: no — catches one kind of network failure, since a deadline can end the wait before a retry. A network error or a 503 leaves open whether PostHog took the send, so a site that falls back to another provider on a failed send stays.
  • sendgrid/mapping.md: a .NET site whose SendGrid client retries moves, and the retry setup only that client used goes; the detection passage behind the old stay rule is cut. A client that ReliabilitySettings set up, with retries above 0 and no HttpClient passed in, as services.AddSendGrid passes one, threw once its last retry failed, so its site throws PostHog's error for a status in that client's RetriableServerErrorStatusCodes, 500, 502, 503 and 504 unless its options set others, and its catch (TimeoutException) becomes catch (OperationCanceledException) only when the solution shows the site's deadline ends before the HttpClient's own Timeout; otherwise the site stays. A RetryDelegatingHandler added by hand to an injected HttpClient with retries above 0 throws the same way; set to 0, or to a count the agent can't read, it keeps the site on SendGrid. A handler that does more than retry, such as a fallback that answers for a send its retries failed, keeps the site on SendGrid, and the reason names the handler. Before moving any .NET site, the agent traces every handler on the client's HttpClient, from AddHttpClient, ConfigureHttpClientDefaults and any factory that builds it, and the site stays unless the solution shows the whole chain and what each handler does; the reason names the handler or the registration where tracing stopped. A RetryDelegatingHandler added by hand to an injected HttpClient throws like ReliabilitySettings. The Node.js quota_limited check on err.code === 429 and the HTTP paragraph's "no longer retries" note are cut. Ruby, PHP, Go and .NET keep each status branch but one on 429, which Step 7 removes. The Go MakeRequestRetry sentence is cut: a 429 left after the SDK's retries takes the path of every other error.
  • resend/mapping.md: "The retry delay" no longer keeps a deadline site's own wait and retry. Python's RateLimitError, Ruby's RateLimitExceededError and Go's rate-limit check no longer become a check of status 429: Step 7 removes them, and so does a Node.js branch on rate_limit_exceeded, except work besides its retry, which stays. The .NET and Elixir name examples use validation_error instead of the rate limit.
  • Both mapping.md files, suppressed recipients: "throws, retries, or reports a failure" drops "retries".
  • rewrite-go.md and rewrite-dotnet.md, "A deadline": zero retries on a deadline send only when the code never retried the request, as Step 7 says, and the deadline passed to the send is the site's deadline over the whole send, never one the code set for each try.
  • 7-rewrite.md "Write down" and 9-report.md: each retry the move removed is noted under the sites table and listed in the report, next to the opens-off and click notes from main.

Known issues

  • context/skills/migrate-email/references/sendgrid/mapping.md:156: a SendGrid .NET site whose ReliabilitySettings retried a transient 500, 502 or 504 now fails on the first such answer from PostHog, because every PostHog SDK retries only a network error, 409 idempotency_key_in_flight, a 429 other than quota_limited, and 503. This follows Michael's 2026-11-17 decision that the app's own retry goes and the SDK retries what a retry can fix, kept in d-2deba2ad.
  • context/skills/migrate-email/references/sendgrid/mapping.md:156: no trial covers a .NET solution whose handler or HttpClient factory comes from a referenced assembly. The rule keeps such a site on SendGrid, read from the text; trial H covers only a handler in the solution whose behaviour can't be read. No trial covers a RetryDelegatingHandler added by hand to an injected HttpClient. The .NET timeout trial site, with ReliabilitySettings, a 5-second token and catch (TimeoutException), moved at 4bed508 and now stays on SendGrid, because the solution can't show its HttpClient's timeout; no run at c400052 shows that stay, or tells it from a site whose token provably ends first.
  • context/skills/migrate-email/references/3-inventory.md:27 and context/skills/migrate-email/references/7-rewrite.md:32: no trial runs a site that falls back to another provider when its send fails. Steps 3 and 7 keep it on the provider; the 503 after queueing is read from PostHog's code.
  • context/skills/migrate-email/references/7-rewrite.md:127: no trial runs a 429 branch that does other work besides its retry against rate_limited and quota_limited; the rule is read from the text.
  • context/skills/migrate-email/references/7-rewrite.md:118: the stay rules for a retry that waits past 30 seconds or does work on each failed try sit in Step 7, after the client setup, so an agent can install or upgrade the PostHog SDK for a site it then keeps on the provider. The site's send is untouched; only the dependency is extra.
  • These trial gaps and the setup order are accepted by Michael at c400052 as d-2a5e8d62, with no further commits.

How it was tested

  • The lane gate on c400052, 72b4041, 8b194e7, ffb093a and 4bed508 (with main merged after feat(migrate-email): map provider tracking settings to tracking and sensitive #486): vitest run scripts, 400 tests, bun run test:skills, bun run test:plugins and node scripts/build.js pass, with a clean tree after the build. Both bundles carry the tracking text, the opens-off rule and the opens-first sentence from main, and this PR's retry text.

  • Shadow diff from main (ee1402d) to the head, c400052, rebuilt fresh on both sides: only 3-inventory.md, 7-rewrite.md, 9-report.md, the provider mapping.md, rewrite-go.md and rewrite-dotnet.md of the migrate-email-resend and migrate-email-sendgrid bundles, and manifest.json, change; every other bundle is byte-identical. Saved in proof storage.

  • Two trial outcomes below change with 72b4041 and c400052, read from the text, not re-run: the SendGrid .NET site with ReliabilitySettings, one 5-second token and catch (TimeoutException) now stays on SendGrid, because SendGrid builds its HttpClient and the solution can't show that client's timeout. The Resend Python site whose RateLimitError branch returns False for a scheduler to try again next hour now stays on Resend, because its resend waits past 30 seconds. Both now leave the old send untouched; the other rows hold.

  • Live run at the head, 4bed508: the built bundle ran with headless Claude Code (Opus 5.5, high effort) on every trial site the changes since 97cc1fd reach. No retry library, loop, wait or 429 branch is left around any moved send, and each moved code send also carries the sensitive flag and opens-off tracking from main:

    Site Retry before Retry after
    SendGrid .NET client with ReliabilitySettings client retries, 3 times none; it moved and throws for 500, 502, 503, 504 and network_error
    SendGrid .NET client from services.AddSendGrid with ReliabilitySettings none: the settings never ran none; it moved and logs every refused send
    SendGrid .NET client with ReliabilitySettings, one 5-second token, and catch (TimeoutException) client retries, 3 times none; it moved, the token goes to the send, and the catch is OperationCanceledException
    SendGrid Go, a loop of 3 tries under one 5-second context, with a net.Error Timeout() check 3 tries unchanged: at ffb093a it stays on SendGrid, moves: no — catches one kind of network failure
    SendGrid HTTP in Python, a urllib3 Retry with its default methods and timeout=5 the adapter retries a failed connection none; it moved with a 5-second email client and the SDK's retries
    Resend Python, except RateLimitError logs and returns False rate-limit branch on 429 none; it moved, with no separate 429 branch

    At ffb093a, after final read 9 and rounds 2 to 4, five more trials ran with the built bundle (Opus 5.5, high), each moved send against a fake PostHog:

    Site Result
    Resend Node.js, a rate_limit_exceeded branch returning its own result moved, branch gone; a lasting 429 rate_limited makes 3 requests and returns the ordinary failure result, a 200 succeeds
    SendGrid .NET, a Polly retry plus a fallback answering 202 on its HttpClient stays on SendGrid, the reason names the fallback handler, code unchanged
    SendGrid Go, the 3-try loop above stays, moves: no — catches one kind of network failure
    SendGrid Go, no loop, one 5-second context, a Timeout() branch moved with MaxRetries: posthog.Ptr(0); a 7-second answer hits the branch at 5 s with 1 request, a 503 takes 1 request
    SendGrid .NET, ReliabilitySettings, one 5-second token, catch (TimeoutException) moved, catch is OperationCanceledException; a slow send is caught at 5 s, a 500 throws after 1 request, a 503 after 3

    At 94be42b, after Michael's d-2deba2ad, two more trials ran, carried to 5e3ce3a, which only widens the .NET stay rule:

    Site Result
    SendGrid Node.js, a hand-written loop of 3 tries with a 500 ms sleep, logging welcome email failed and returning { ok: false } after the last moved; loop, sleep and counter gone; through the caller a lasting 503 makes 3 requests, logs the line and returns { ok: false }, a 400 takes 1 request, a 200 returns { ok: true }; the Step 9 report names the removed loop
    SendGrid .NET, an AuditHandler added by ConfigureHttpClientDefaults that calls an IAuditSink with no implementation in the solution stays, the reason names AuditHandler, no file changed

    Earlier, at 4bed508, the three .NET sites ran against a fake PostHog: 3 requests for a 503 and a rate-limited 429, 1 for every other answer; the standalone client's sites throw for a 500 and a 503, the AddSendGrid site logs every status, and the token site catches the cancellation at 5 seconds. Saved in proof storage as the live run.

  • Live run at 97cc1fd, for the 429 and .NET status changes: the same harness on every trial site they reach:

    Site Retry before Retry after
    Resend Python, except RateLimitError logs and returns False so a scheduler tries again next hour rate-limit branch on 429 none; it moved, with no separate 429 branch around emails.send
    SendGrid .NET client with ReliabilitySettings client retries, 3 times none; it moved and throws for 500, 502, 503, 504 and network_error
    SendGrid .NET client with ReliabilitySettings and one 5-second token on SendEmailAsync client retries, 3 times none; it moved, the token goes to the send, and it throws for the same statuses

    Each moved site then ran against a fake PostHog: 3 requests for a 503 and a rate-limited 429, 1 for every other answer. The Resend site raises for every error after the SDK's tries; where the old code returned False on a rate limit, it now raises. Both .NET sites throw for a 500 or a 503 and log every other refused send, as their old clients did, and the token cancels at 5 seconds. SendGrid's default list was read from SendGrid 9.29.3 at run time. One real send through the moved Resend site arrived in the test inbox, sent from a verified sender: the fixture's own sender is not verified in the test project, and PostHog refused it. The other sites below are Node.js or Python SendGrid sites, a Resend Node.js site, or sites that stay before any status branch, so 97cc1fd does not reach them.

  • Live run at 4ecadc2: the built bundle ran with headless Claude Code (Opus 5.5, high effort) on every trial site the deadline rules can reach, plus the p-retry site for the real send. The account steps were taken as done, and nobody answered questions. A scan of each codebase's source then looked for a retry library or decorator, a loop, a wait, or a 429 branch in each file that sends:

    Site Retry before Retry after
    SendGrid Node.js, p-retry around sgMail.send p-retry, 3 retries none; it moved
    SendGrid HTTP in Node.js, a hand-written loop that waits and resends on 429, no deadline wait and resend on 429, 5 attempts none; it moved
    SendGrid HTTP in Python, a urllib3 Retry adapter and timeout=5 the adapter retries the POST none; it moved (it stayed on the base) with a 5-second email client and the SDK's retries
    SendGrid .NET client with ReliabilitySettings and one 5-second token on SendEmailAsync client retries, 3 times none; it moved (it stayed on the base), the token goes to the send, and the SDK keeps its retries
    SendGrid HTTP in Node.js, a loop with one AbortSignal.timeout(5000) across 3 tries 3 tries within 5 seconds unchanged; it stays on SendGrid, as on the base
    SendGrid Python, tenacity stop_after_delay(10) tries for 10 seconds unchanged; it stays on SendGrid
    SendGrid .NET, a loop with a new 5-second token for each try, whose caller catches only the timeout 3 tries unchanged; it stays on SendGrid: it catches one kind of network failure
  • Behaviour against a fake PostHog, at 4ecadc2, for the sites above it: each moved site's own module ran its send once per answer: 200, 400, 422 recipient_suppressed, 500, 503, 429 rate_limited, 429 quota_limited, and a 7-second answer for the deadline sites. Node.js ran on this machine, Python in a python:3.12-slim container, and the .NET ReceiptMailer.cs compiled in the .NET 8 SDK image. Every site made 3 requests for a 503 and a rate-limited 429, which are the SDK's own tries, and 1 for every other answer, so no app retry is left. Each caller then threw, raised, or logged and went on exactly as its old code did after its last retry. The slow answer ended the Python site after 3 tries of 5 seconds with network_error, and the .NET site at 5 seconds with TaskCanceledException, as its old token did.

  • Real send, at 4ecadc2: the moved p-retry app ran with a project secret key, and one POST /signup sent through PostHog's US Cloud. The email arrived in the test inbox in the same second. Saved in proof storage as the live run.

  • Earlier live run, at 393fefe, on eight codebases: sites without a deadline, whose rewrite nothing since has changed, with the same behaviour runs and a delivered real send:

    Site Retry before Retry after
    SendGrid Node.js, p-retry around sgMail.send p-retry, 3 retries none
    SendGrid Python, tenacity @retry on the send tenacity, 4 attempts, exponential wait none
    SendGrid HTTP in Node.js, a hand-written loop that waits Retry-After and resends on 429 wait and resend on 429, 5 attempts none
    SendGrid .NET client with ReliabilitySettings client retries, 3 times none; it moved (it stayed on the base)
    Resend Node.js, async-retry around resend.emails.send async-retry, 3 retries none
    Control: SendGrid Node.js with no retry none none added
    SendGrid HTTP in Python, a requests session with a urllib3 Retry adapter and timeout=5 the adapter retries the POST none; it moved (it stayed on the base) with a 5-second email client and the SDK's retries
    SendGrid Node.js in a BullMQ worker: re-queue with a delay on 429, attempts: 5 on the job re-queue only to retry none; the job's attempts stays

    Every site moved, and each report names the removed retry. The first trial, at efbd483, gave the .NET site no throw for a server error, which round 1 caught.

Agent context
Question Answer Who Where
What the SDKs retry The skill names them: a network error, a 429 other than quota_limited, and 503, never 500, 502 or 504 👤 Human transactional email conductor, 2026-11-17
Retried answers where the SDKs differ Name what all eight share, adding 409 idempotency_key_in_flight; posthog-rs alone also retries 429 upload_quota_exceeded, left out 👤 Human transactional email conductor, 2026-11-18
An app's own retry around the send Remove it: the SDK already retries what a retry can fix 👤 Human transactional email conductor, 2026-11-17
The judge's 429 re-queue, hand-added RetryDelegatingHandler and 503 items Fix 1 and 3 with a clause each, check 2 in posthog's code (it held: line 32 fix), one fixes round, blockers fixed in 9bf50de and 8b194e7
The judge's remaining items at c400052: untried stay rules, the setup order, and its 429 escalation Accept as a judge proof gap, no more commits, all listed under Known issues 👤 Human, d-2a5e8d62 transactional email conductor, 2026-11-19
The judge's long Retry-After, per-try wrapper work and Resend rate_limit_exceeded items Two stay clauses on Step 7's retry rule and one Resend clause in 72b4041, one last fixes round, no further rounds 👤 Human, d-d3f3bd05 transactional email conductor, 2026-11-19
The merge judge's .NET and retry-loop items Add the handler-search sentence, run one retry-loop trial, keep the 500, 502 and 504 change under Known issues, one fixes round 👤 Human, d-2deba2ad transactional email conductor, 2026-11-19

Involvement: no grilling rounds, no prototypes. Review round 1 (GPT-6.1 Sol), final reads 1 to 9 and fix rounds 2 to 4 (GPT-6.2 Sol, two lenses, two families), a reviewer's change request, a second reviewer's tool-assisted comments, and ReviewHog: every finding fixed, except one declined with a reason, and a format change left for a follow-up. Final read 9 covers 4ecadc2 to 4bed508 under map #315's standing review rule; round 3's Go blocker, the third on that check, was fixed by cutting scope back to main's rule, and round 4's only finding was this description. Round 5, the one fixes round Michael chose, found a blocker in the new .NET sentence, fixed in 5e3ce3a so an untraceable handler chain keeps the site, and asked for a referenced-assembly trial, listed under Known issues. Round 6, on 4d855fb's 429, RetryDelegatingHandler and 503 clauses from Michael's d-ffbf97bb, found three blockers and a major, all fixed in 9bf50de and 8b194e7: the failover stay moves to Step 3, the kept 429 work gets one path, a .NET timeout catch without a kept deadline stays, and SendGrid's own handler is no unreadable one. Its trial asks are under Known issues. Round 7, the last fixes round Michael chose in d-d3f3bd05, reviews those fixes and 72b4041's stay clauses for a retry that waits past 30 seconds or works on each try. It found three blockers, fixed in c400052: a .NET timeout catch moves only when the solution shows the deadline ends before the HttpClient's own timeout, a hand-added SendGrid handler set to 0 retries keeps its site, and the retry stay rules are decided before the 429 rule. Its two trial majors are under Known issues, and no further round runs. Proof captured: shadow diff and live run (trials on generated codebases at ffb093a, 4bed508 and before, their moved sends run against a fake PostHog, and one real send).

🤖 Generated with Claude Code

Silthus and others added 4 commits November 17, 2026 09:56
The SDKs retry, with the same idempotency key, only the answers that
mean PostHog didn't take the send. The skill now removes an app's own
retry around the send instead of keeping or rebuilding it, moves .NET
SendGrid sites whose client retried, and keeps the SDK's retries for a
deadline site that retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e .NET throw

A deadline send in Go or .NET whose request retried no longer takes the
rewrite file's zero retries. A .NET SendGrid client that retried threw
once its last retry failed, so its moved site throws PostHog's error
for a server error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Only SendGrid's own retry handler threw once its retries ran out; a
retry handler on the HttpClient keeps what it did. A deadline the app
set for each try becomes the email client's timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job framework's rerun is left alone, as on the provider; only an
idempotency key the site sets keeps a rerun from sending twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 1ba742d: 1 Should fix.

See this review and its findings in PostHog Review. Comment @posthog review --deep for a Deep review. A Deep review takes longer and looks deeper than a Standard review.

Silthus and others added 5 commits November 17, 2026 10:47
… can

A retry loop that shares one deadline across its tries, such as one
AbortSignal over every fetch, keeps that deadline only where the send
takes it (Go's context, .NET's token) or a timer stays around the send.
Elsewhere the SDK can't keep it, and the site stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…'t keep stays

Name the case in the stay sentence and say such a deadline is never the
client's timeout, with p-retry maxRetryTime and tenacity stop_after_delay
as examples.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A timer around a send whose SDK keeps retrying rejects while the send
can still deliver, so only a send that takes the deadline keeps it.
State the stay rule once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n .NET too

A deadline set for each try is the email client's timeout, so its
timeout comes back as network_error, never as the token's
OperationCanceledException that the .NET bullet describes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uage bullets

Say in the exception's lead-in that a deadline set for each try is never
one the send takes, so an agent never applies the Go or .NET bullet to
it. The last bullet goes back to its base wording.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 4ecadc2: no findings.

@Silthus
Silthus marked this pull request as ready for review November 17, 2026 11:57
@Silthus
Silthus requested a review from a team as a code owner November 17, 2026 11:57

@sarahxsanders sarahxsanders left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once one of #486 and #541 lands, update the other and check that the built Resend and SendGrid skills include both the tracking and retry changes. The two PRs conflict in Steps 7 and 9.

- In every other SDK the deadline is the client's timeout, and its timeout is a `network_error` like any other network failure: a branch on it stays.
- A branch on the provider's answer to an invalid send, such as a `400` or a `422`, becomes a branch on PostHog's error name `validation_error`, never on a status code: PostHog's statuses for invalid sends differ from the provider's, and `recipient_suppressed` is a `validation_error` with status `422`. Handle `recipient_suppressed` before that branch, as the provider's `mapping.md` says.
- A branch on the provider's rate limit, such as a retry, a wait, or a re-queue on status `429`, becomes a branch on PostHog's status code `429` with a code other than `quota_limited`. PostHog answers a used-up sending quota with status `429` and the code `quota_limited`, which no retry fixes: it takes the path of every other error.
- A branch on the provider's rate limit, such as a retry, a wait, or a re-queue on status `429`, goes, as the bullet on retries says. A `429` that is left after the SDK's retries takes the path of every other error, as `quota_limited` does.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step 7 intentionally sends a final 429 through the generic error path, but the Resend mapping still tells the agent to keep a separate 429 check in Python, Ruby, and Go. Update those examples to follow Step 7 so the agent gets one instruction.

…ppings

Step 7 removes a branch on the provider's rate limit, and a 429 left
after the SDK's retries takes the path of every other error. The Resend
Python, Ruby and Go examples and the SendGrid status-branch rule still
kept a 429 check, and the .NET and Elixir name examples used the rate
limit. Align them with Step 7 so the agent gets one instruction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus

Silthus commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@PostHog review

@posthog

posthog Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

@Silthus PostHog Review is reviewing this pull request.

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed d96e9a0: 1 Should fix.

Silthus and others added 2 commits November 18, 2026 17:56
…nt retried

SendGrid's RetryDelegatingHandler throws only for a status in
RetriableServerErrorStatusCodes. A client that left 500 out of that list
got a 500 back without a throw, so its moved site must not throw for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ReliabilitySettings retries 500, 502, 503 and 504 unless its options set
others (read from SendGrid 9.29.3 at run time). Without the list, a trial
agent guessed 503 and 504 and let a 500 or 502 through.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus

Silthus commented Nov 18, 2026

Copy link
Copy Markdown
Contributor Author

@PostHog review

@posthog

posthog Bot commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

@Silthus PostHog Review is reviewing this pull request.

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 97cc1fd: no findings.

@gewenyu99 gewenyu99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🥸 Reviewed by NotVincent, totally not Vincent. The real Vincent will review it separately. Probably slop, please disregard.

- **Python.** PostHog's SDK raises too. When the same `try` also holds a SendGrid call that stays, such as a suppression call, keep each `except HTTPError` handler for that call's errors, and add the PostHog exception's handling next to it.
- **Ruby, PHP, Go, and .NET.** The code read SendGrid's status code. Read PostHog's error the way the rewrite file says under "The answer", and keep each branch: a status of 400 or more becomes "there is an error", a branch on `400` reads PostHog's error name `validation_error`, and a branch on another status, such as 429, reads PostHog's status code. Their SendGrid clients don't raise on a refused send: a site that never read the status went on after one, and only a network failure raised (Ruby, PHP, .NET) or set `err` (Go). A .NET client whose options set `HttpErrorAsException` is the exception: it threw on a refused send, so its sites throw PostHog's exception the way they threw before, and the next sentence does not apply to them. Such a site logs PostHog's error and goes on, and raises or returns it when its code is `network_error`, as Step 7 says. A Go site that sent through `MakeRequestRetry` or `MakeRequestRetryWithContext` also got `err` when SendGrid still answered 429 after its last retry (`rate limit retry exceeded`): it also returns PostHog's error when its status code is 429, which PostHog's send answers only once its own retries are used up. A .NET site whose SendGrid client retries server errors stays, and the reason is `its client retries server errors`: the client retried a server error, SendGrid's `RetryDelegatingHandler` threw `HttpRequestException` after its last retry, and PostHog's send does neither. The client retries when its options set `ReliabilitySettings` with `MaximumNumberOfRetries` above 0 and it makes its own `HttpClient`, or when the code adds a handler that retries to the `HttpClient` the client gets, the one it passes to the client or the one of the client `AddSendGrid` registers: `RetryDelegatingHandler` with `MaximumNumberOfRetries` above 0, a Polly policy, or a resilience handler such as `AddStandardResilienceHandler`. `ConfigureHttpClientDefaults` adds its handlers to every client from .NET's factory, and often sits in another project, such as an Aspire app's service defaults: search the whole solution for it before a site whose `HttpClient` comes from .NET's factory moves, an `AddSendGrid` site or one given `CreateClient`'s client alike. `AddSendGrid` gives the client an `HttpClient` from .NET's factory, so the `ReliabilitySettings` in its options retry nothing by themselves. When the code can't tell what a handler it adds there does, or how many times it retries, the site stays too.
- PostHog refuses a send to a suppressed address with `recipient_suppressed`, where SendGrid took it and dropped it later. When the site throws, retries, or reports a failure on any error, handle `recipient_suppressed` before that branch the way the site handled a send that went out: log it and go on. Note each site that got this branch in the report.
- **Ruby, PHP, Go, and .NET.** The code read SendGrid's status code. Read PostHog's error the way the rewrite file says under "The answer", and keep each branch but one on `429`, which Step 7 removes: a status of 400 or more becomes "there is an error", a branch on `400` reads PostHog's error name `validation_error`, and a branch on another status reads PostHog's status code. Their SendGrid clients don't raise on a refused send: a site that never read the status went on after one, and only a network failure raised (Ruby, PHP, .NET) or set `err` (Go). A .NET client whose options set `HttpErrorAsException` is the exception: it threw on a refused send, so its sites throw PostHog's exception the way they threw before, and the next sentence does not apply to them. Such a site logs PostHog's error and goes on, and raises or returns it when its code is `network_error`, as Step 7 says. A Go site that sent through `MakeRequestRetry` or `MakeRequestRetryWithContext` also got `err` when SendGrid still answered 429 after its last retry (`rate limit retry exceeded`): it also returns PostHog's error when its status code is 429, which PostHog's send answers only once its own retries are used up. A .NET site whose SendGrid client retries moves too: remove the retry setup that only that client used, `ReliabilitySettings` in its options or a retry handler on its `HttpClient`, and never touch `ConfigureHttpClientDefaults` or a handler that other clients share. A client that `ReliabilitySettings` set up threw once its last retry failed: SendGrid's `RetryDelegatingHandler` throws `HttpRequestException` for a status in its `RetriableServerErrorStatusCodes` that is left, `500`, `502`, `503`, and `504` unless the options set others, so its site throws PostHog's error for those statuses.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think moved .NET sites that got their SendGrid client from services.AddSendGrid(...) will newly throw on a server error, and standalone sites will lose their catch (TimeoutException), because:

  • AddSendGrid hands an IHttpClientFactory HttpClient to SendGrid's BaseClient, which runs httpClient ?? CreateHttpClientWithRetryHandler(options). ReliabilitySettings never ran for those clients, so their sites never threw on a 5xx, and this rule still makes them throw.
  • RetryDelegatingHandler also passes the request straight through when MaximumNumberOfRetries is 0.
  • Where the handler did run, it caught TaskCanceledException after its last try and threw System.TimeoutException. PostHogClient lets OperationCanceledException through, so a catch (TimeoutException) around the send stops running.

Scope the throw rule to a client built without an injected HttpClient and with MaximumNumberOfRetries above 0, and map a catch (TimeoutException) on those sites to OperationCanceledException.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mapping file should honestly not be a markdown 👁️ Maybe a json/yaml what ever

- When the old code caught only some errors, such as one error class or one status, catch PostHog's error and branch on its `code` or status code for those, and raise or return every other error as before. Several handlers of the provider's error classes on one `try` become one catch of PostHog's error that branches in their order: the first of several catches of the same error type would take every error. Never widen a catch. PostHog reports a network error with the same error type as an API error, with the code `network_error`. When the old catch took only the provider's answers, such as an HTTP error class, and a network failure passed it, raise or return `network_error` the way the old code let a network failure through. When a later handler of the same `try`, such as a broader `rescue` or `except` after it, took that network failure, run that handler's code for `network_error`: a raise inside one handler skips the handlers after it. A catch of every error around the send, such as a bare `except:`, `except Exception`, `rescue => e`, or `catch (\Throwable $e)`, caught network failures before too: it stays as it is, and lets no `network_error` through, here and in the bullet above.
- When the old code told one kind of network failure from another, the site stays: `moves: no — catches one kind of network failure`. That is a branch on the transport's error code or class, such as `err.code === 'ECONNRESET'` or `'ETIMEDOUT'` in Node.js, `requests.Timeout`, `requests.ConnectionError`, or `e.errno` in Python, `Faraday::ConnectionFailed`, `Net::ReadTimeout`, or `Errno::ECONNRESET` in Ruby, or curl's error number in PHP. Every SDK retries network failures and reports what is left as one `network_error`, which doesn't say which kind it was, so the branch would never run. A catch that takes every network failure alike, such as a `catch` around `fetch`, which throws only when no answer arrived, is not such a branch: it becomes a branch on `network_error`, as above. The one exception is the timeout of a deadline the site keeps, where the send takes the deadline itself:
- When the old code told one kind of network failure from another, the site stays: `moves: no — catches one kind of network failure`. That is a branch on the transport's error code or class, such as `err.code === 'ECONNRESET'` or `'ETIMEDOUT'` in Node.js, `requests.Timeout`, `requests.ConnectionError`, or `e.errno` in Python, `Faraday::ConnectionFailed`, `Net::ReadTimeout`, or `Errno::ECONNRESET` in Ruby, or curl's error number in PHP. Every SDK retries network failures and reports what is left as one `network_error`, which doesn't say which kind it was, so the branch would never run. A catch that takes every network failure alike, such as a `catch` around `fetch`, which throws only when no answer arrived, is not such a branch: it becomes a branch on `network_error`, as above. The one exception is the timeout of a deadline the site keeps, where the send takes the deadline itself, never one the code set for each try:
- Go: `Send` ends at the context's deadline with a `*posthog.EmailError` that unwraps to the request's error, so `errors.Is(err, context.DeadlineExceeded)`, and a `net.Error` read with `errors.As` whose `Timeout()` is true, still match.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think retrying Go sites that check errors.As(err, &netErr) && netErr.Timeout() will miss a context deadline that fires during the wait before a retry, because:

  • A Go deadline send that keeps the SDK's retries runs posthog-go's backoff under the caller's context.WithTimeout.
  • When the first try fails with a net.Error that is no timeout, such as ECONNREFUSED, and the context ends during the wait, posthog-go joins them with errors.Join(e.cause, context.DeadlineExceeded).
  • errors.As(err, &netErr) finds the first try's *url.Error first, so netErr.Timeout() is false.
Suggested change
- Go: `Send` ends at the context's deadline with a `*posthog.EmailError` that unwraps to the request's error, so `errors.Is(err, context.DeadlineExceeded)`, and a `net.Error` read with `errors.As` whose `Timeout()` is true, still match.
- Go: `Send` ends at the context's deadline with a `*posthog.EmailError` that unwraps to `context.DeadlineExceeded`, so `errors.Is(err, context.DeadlineExceeded)` still matches. When `Send` keeps retries and the deadline ends a retry wait after a non-timeout `net.Error`, `errors.Join` puts the first `net.Error` ahead of `context.DeadlineExceeded`, so rewrite a `net.Error` `Timeout()` check to also match `errors.Is(err, context.DeadlineExceeded)`.

- On a self-hosted instance, also set the app host, under the name the rewrite file gives, to the instance's app URL. The email send goes only to the app host, and without it the send fails with `missing_app_host`.
- Put the email client in its own server-only module, next to the provider's client, and use it where the sites send.
- A site whose request had its own deadline keeps it, as the provider's `mapping.md` says under "HTTP": the send gets that deadline, and no retries, since the request never retried. Every SDK otherwise waits up to 15 seconds for each request and retries twice, which can run past the site's deadline. The rewrite file names the timeout and retry options under "A deadline". Where the send takes the deadline itself, as Go's context and .NET's cancellation token do, pass it there; the SDK still ends each request at its own timeout, so a deadline longer than that also needs a client of its own with the timeout set to it. Otherwise each distinct deadline gets an email client of its own with that timeout, in the same module, and its sites send through it. Sites without a deadline keep the first client. Never give a site a deadline it didn't have, or change one it had. A request with its own deadline whose HTTP client the code also set up to retry a POST did retry, and the send that keeps the deadline takes no retries: the site stays, `moves: no — the PostHog SDK can't keep the request's retries`. Such setups include `axios-retry`, which by default retries a network error on any method, a `requests` session adapter with a urllib3 `Retry` or a number as its `max_retries`, such as `HTTPAdapter(max_retries=3)`, which by default retries a failed connection on any method, or Faraday's `retry` middleware with `:post` in its `methods:` or a `retry_if`. Read the options the code gives the setup, such as `axios-retry`'s `retries` and `retryCondition`, or `Retry`'s `total`, `connect` and `allowed_methods`, where a number stands for `total`, to tell whether it retries the POST. A retry setup that skips a POST, such as Faraday's `retry` with its default methods or a setup with no retries left, never retried the send, and the site keeps its deadline with no retries. Only a limit the code sets is the site's own deadline: a library's default, such as `httpx`'s 5 seconds or `ky`'s 10, is not one, like a provider SDK's default, and such a site sends with the PostHog SDK's own timeout and retries. The SDK keeps a deadline only of the kind its timeout is. Every SDK's timeout limits the whole request, except Python's and Ruby's, which limit each wait on the network, as their rewrite files say. A limit of another kind is one the SDK can't keep: axios's `timeout` in Node.js ends a request only after that long without data, and `ky`'s `timeout` ends once the answer's headers arrive. Every SDK has one timeout for connecting and reading alike, so a request that limits only connecting, or limits connecting and reading apart, such as Faraday's `open_timeout`, Guzzle's `connect_timeout`, `requests`' `timeout=(1, 10)`, or an `httpx.Timeout` with its own `connect`, has a deadline the SDK can't keep, unless the rewrite file says under "A deadline" how its client keeps such a limit, as Go's does. When the SDK can't hold the deadline, as there or with a deadline that is not whole seconds in Rust, the site stays: `moves: no — the PostHog SDK can't keep the request's deadline`.
- A site whose request had its own deadline keeps it, as the provider's `mapping.md` says under "HTTP": the send gets that deadline, and no retries when the code never retried the request. Every SDK otherwise waits up to 15 seconds for each request and retries twice, which can run past the site's deadline. The rewrite file names the timeout and retry options under "A deadline". Where the send takes the deadline itself, as Go's context and .NET's cancellation token do, pass it there; the SDK still ends each request at its own timeout, so a deadline longer than that also needs a client of its own with the timeout set to it. Otherwise each distinct deadline gets an email client of its own with that timeout, in the same module, and its sites send through it. Sites without a deadline keep the first client. Never give a site a deadline it didn't have, or change one it had. When the code retried the request, around the send or through its HTTP client, such as `axios-retry`, a urllib3 `Retry` adapter, or Faraday's `retry` with `:post`, the send keeps the SDK's own retries, with no zero retries from the rewrite file's "A deadline", and that retry setup goes when nothing else uses it. A deadline the code set for each try is the email client's timeout, never one passed to the send. A deadline over all the tries, such as one `AbortSignal` across them, p-retry's `maxRetryTime`, or tenacity's `stop_after_delay`, is never the client's timeout: the SDK keeps it only where the send takes it, as Go's context and .NET's token do. Only a limit the code sets is the site's own deadline: a library's default, such as `httpx`'s 5 seconds or `ky`'s 10, is not one, like a provider SDK's default, and such a site sends with the PostHog SDK's own timeout and retries. The SDK keeps a deadline only of the kind its timeout is. Every SDK's timeout limits the whole request, except Python's and Ruby's, which limit each wait on the network, as their rewrite files say. A limit of another kind is one the SDK can't keep: axios's `timeout` in Node.js ends a request only after that long without data, and `ky`'s `timeout` ends once the answer's headers arrive. Every SDK has one timeout for connecting and reading alike, so a request that limits only connecting, or limits connecting and reading apart, such as Faraday's `open_timeout`, Guzzle's `connect_timeout`, `requests`' `timeout=(1, 10)`, or an `httpx.Timeout` with its own `connect`, has a deadline the SDK can't keep, unless the rewrite file says under "A deadline" how its client keeps such a limit, as Go's does. When the SDK can't hold the deadline, as there or with a deadline that is not whole seconds in Rust, the site stays: `moves: no — the PostHog SDK can't keep the request's deadline`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think an agent that follows "A deadline" in the language rewrite guides, or that trusts any HTTP retry adapter it sees, will set the wrong timeouts and retries, because:

  • The Go and .NET guides still say to pass the deadline's context or token and set MaxRetries to 0 for every deadline site. This rule keeps the SDK's retries when the code retried, and puts a per-try deadline on the client's timeout.
  • urllib3's Retry leaves POST out of allowed_methods by default, and axios-retry's default condition skips POST on a 5xx. A site with such an adapter never retried its send, yet this rule gives it three SDK tries on a single-try deadline.

Bring the Go and .NET "A deadline" sections in line with this rule, and keep the check that the HTTP client's retry setup really retried POST before the send keeps the SDK's retries.

- An error's name is a coarse category with Resend's error names: `validation_error`, `missing_api_key`, `invalid_permission`, `not_found`, `concurrent_idempotent_requests`, `invalid_idempotent_request`, `rate_limit_exceeded`, `monthly_quota_exceeded`, `service_unavailable`, or `application_error`. Its code is PostHog's own, such as `recipient_suppressed`, `sender_not_verified`, or `quota_limited`: branch on the code. A send to a suppressed address fails with the code `recipient_suppressed` and the name `validation_error`.
- The answer carries no response headers. An error carries the retry delay PostHog named, when it named one, under the name the rewrite file gives under "The answer".
- Every SDK retries network errors and rate limits by itself, with the same idempotency key. A network error that is left after the retries comes back as the same error type as an API error, with the code `network_error`.
- PostHog owns delivery. A send that succeeds is queued, not delivered: PostHog retries the delivery itself, and the answer says nothing about it. Every SDK retries, with the same idempotency key, only the answers that mean PostHog didn't take the send: a network error, a rate limit (`429` with a code other than `quota_limited`), and `503`. Every other error is final. A network error that is left after the retries comes back as the same error type as an API error, with the code `network_error`.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think HTTP-boundary tests that Step 8 rewrites for a 409 idempotency_key_in_flight will fail on a retry request they don't expect, because:

  • The SDKs also retry 409 when the code is idempotency_key_in_flight, and don't retry 429 upload_quota_exceeded. I checked posthog-go's emails_error.go, and haven't checked the other seven.
  • Step 8 uses this list to decide which mocked statuses need three stubbed answers and which need one.
Suggested change
- PostHog owns delivery. A send that succeeds is queued, not delivered: PostHog retries the delivery itself, and the answer says nothing about it. Every SDK retries, with the same idempotency key, only the answers that mean PostHog didn't take the send: a network error, a rate limit (`429` with a code other than `quota_limited`), and `503`. Every other error is final. A network error that is left after the retries comes back as the same error type as an API error, with the code `network_error`.
- PostHog owns delivery. A send that succeeds is queued, not delivered: PostHog retries the delivery itself, and the answer says nothing about it. Every SDK retries, with the same idempotency key, only the answers that mean PostHog didn't take the send: a network error, an idempotency key still in flight (`409` with the code `idempotency_key_in_flight`), a rate limit (`429` with a code other than `quota_limited` or `upload_quota_exceeded`), and `503`. Every other error is final. A network error that is left after the retries comes back as the same error type as an API error, with the code `network_error`.

Silthus and others added 5 commits November 18, 2026 19:05
… list to what the SDKs do

- SendGrid .NET: only a client that built its own HttpClient with retries above 0 threw; AddSendGrid and an injected HttpClient never ran ReliabilitySettings. Its catch (TimeoutException) becomes catch (OperationCanceledException).
- Go: a deadline that ends the wait before a retry joins the last try's net.Error ahead of context.DeadlineExceeded, so a Timeout() check also checks errors.Is.
- Retried answers name 409 idempotency_key_in_flight, which all eight SDKs retry.
- Restore the check that an HTTP client's retry setup covers the POST.
- Go and .NET "A deadline": zero retries only when the code never retried.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keeps #486's tracking text, opens-off rule and opens-first sentence, and
#541's retry text. Steps 7 and 9 list both the opens-off sites and the
removed retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s on the context, unknown outcome of a network error

- sendgrid/mapping.md: drop the Go MakeRequestRetry sentence that returned PostHog's error on any 429; a 429 left after the SDK's retries takes the path of every other error, as Step 7 says.
- rewrite-go.md and rewrite-dotnet.md: the context or token carries a deadline over the whole send; a deadline set for each try is EmailRequestTimeout.
- 7-rewrite.md: a network error leaves open whether PostHog took the send.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s answer, no Resend rate-limit branch, .NET fallback handlers stay

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the SDK's retries stays

A deadline can end the wait before a retry, so PostHog's error can't tell
the last try's timeout from the deadline. Keep main's rule for a send
with no retries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silthus and others added 8 commits November 19, 2026 05:32
…before moving it

A handler registered elsewhere, in AddHttpClient or
ConfigureHttpClientDefaults, can answer for a failed send. When its
behaviour can't be read, the site stays on SendGrid (decision d-2deba2ad).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er chain is traced

A factory or handler from a referenced assembly can install a fallback the
solution doesn't show, so the site stays and the reason names where
tracing stopped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A 429 branch that does more than retry the send keeps that work.
- A RetryDelegatingHandler added by hand to an injected HttpClient threw
  like ReliabilitySettings, so its site throws the same way.
- A 503 can come back after PostHog queued the send (Django's CDP call
  timing out after the insert, or async intake without its ack), so a
  site that falls back to another provider on a failed send stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…429 branch, .NET timeout catch

- Step 3: a branch whose failed send falls back to another provider
  stays, so the stay rule applies before Step 7. A 409 still in flight
  leaves the outcome open too.
- Step 7: a 429 branch that does more than retry keeps only that work,
  on a 429 other than quota_limited, then goes on as it did.
- SendGrid .NET: catch (TimeoutException) becomes
  catch (OperationCanceledException) only when the site keeps a deadline
  over the whole send; otherwise it caught the client's own timeout, which
  PostHog reports as network_error, so the site stays. SendGrid's own
  RetryDelegatingHandler is no unreadable referenced-assembly handler.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…9 branch stays

A branch that only gives its caller a rate-limit answer goes with the
retry, as before. Work besides it, such as recording a throttle event,
stays on a 429 other than quota_limited and then takes the ordinary
error path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… stays

- Step 7: a retry that can wait more than 30 seconds stays, since every
  SDK gives up at once on a Retry-After over 30 seconds; so does a retry
  that does work on each failed try, which the SDK's retries can't run.
- Resend mapping: a rate_limit_exceeded branch goes, except work besides
  its retry, as Step 7 says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ts before moving

- Step 7: decide whether the retry rule keeps a site, by a wait past
  30 seconds or work on each failed try, before the 429 bullet; a resend
  left to a scheduler counts, a log line is no per-try work.
- SendGrid .NET: catch (TimeoutException) becomes
  catch (OperationCanceledException) only when the solution shows the
  site's deadline ends before the HttpClient's own Timeout. A hand-added
  RetryDelegatingHandler throws only with retries above 0; set to 0, or to
  a count you can't read, it keeps the site on SendGrid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus merged commit b4d7cd9 into main Nov 20, 2026
17 checks passed
Silthus added a commit that referenced this pull request Nov 20, 2026
Brings in #541 (send retries left to PostHog's SDK) and #543. The two
conflicts in 7-rewrite.md "Write down" and 9-report.md's notes list
keep both sides: main's note on each retry the move removed and this
branch's plain-text click notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants