diff --git a/AGENTS.md b/AGENTS.md index 45bd33c7..e7c6faeb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -95,6 +95,7 @@ longer exist — only some keep an alias. |---|---| | `wizard migrate statsig` | `migrate-statsig` | | `wizard migrate resend` | `migrate-email-resend` | +| `wizard migrate sendgrid` | `migrate-email-sendgrid` | **Commands vs. skills:** those audit subcommands **are** skills, promoted to commands via `cli: role: command`. A `role: skill` skill is reachable only via diff --git a/context/skills/migrate-email/config.yaml b/context/skills/migrate-email/config.yaml index 6f9a304f..0e819850 100644 --- a/context/skills/migrate-email/config.yaml +++ b/context/skills/migrate-email/config.yaml @@ -16,3 +16,6 @@ variants: - id: resend display_name: Resend tags: [resend] + - id: sendgrid + display_name: SendGrid + tags: [sendgrid] diff --git a/context/skills/migrate-email/references/sendgrid/account.md b/context/skills/migrate-email/references/sendgrid/account.md new file mode 100644 index 00000000..945c9e13 --- /dev/null +++ b/context/skills/migrate-email/references/sendgrid/account.md @@ -0,0 +1,104 @@ +# SendGrid account + +What the migration needs from the person's SendGrid account. The agent never changes the account: it reads what the person exports or allows. + +**EU subusers.** An app on an EU subuser sends through `api.eu.sendgrid.com`, or sets the client's data residency to `eu`, as `sdk-reference.md` says. For such an app, use `api.eu.sendgrid.com` in every request of this file instead of `api.sendgrid.com`. + +**Subusers.** A send uses the suppression lists, templates, and settings of the account its API key belongs to: the parent account or one of its subusers. SendGrid's `on-behalf-of` header, which a client's subuser option such as `setImpersonateSubuser` in Node.js or `GetRequestSubuser` in Go sets, doesn't work with the Mail Send API, so it never sends for another account. Ask the person which account the app's sending key belongs to, make the read-only key below in that account, and switch to that account in SendGrid for every question and export of this file. When sites send with keys of different accounts, or a site picks its key at run time, such as per tenant, ask the person which account's lists Step 5 imports: the sites that send only with that account's key move, and every other site stays, with the reason `sends for another SendGrid account`. When the person can't say which account a key belongs to, its sites stay, with the reason `SendGrid account not checked`. + +## DNS records + +A domain authenticated at SendGrid carries these records. Keep all of them until the cutover is done. The SendGrid dashboard shows the exact names under Settings, then Sender Authentication. + +| Record | Name | What it does | +|---|---|---| +| `CNAME` | `em`, such as `em1234` | Return path for SendGrid's bounces, and its SPF | +| `CNAME` (DKIM) | `s1._domainkey`, `s2._domainkey` | Signs SendGrid's mail | +| `CNAME` | `url` and a second label of digits | Link branding, only when it is set up | +| `TXT` (DMARC) | `_dmarc` | Optional, shared by every provider on the domain | + +- A domain set up without automated security has an `MX` and an SPF `TXT` record on `em`, and a DKIM `TXT` record at `m1._domainkey`, in place of the CNAMEs. +- PostHog's sender uses its own label for its MAIL FROM, `feedback` by default. Never give PostHog a label SendGrid uses. + +## Read-only key + +The app's own SendGrid key is usually a key with the Mail Send permission only, and it can't read templates or suppressions. When the person chooses the API over the dashboard, ask them to create a restricted key in SendGrid (Settings, then API Keys) with read access to Template Engine and Suppressions only, store it in `SENDGRID_READ_API_KEY` in the git-ignored env file, and write it into the plan as Step 3's plan layout says. Its last use is the suppression import in Step 5. At the end of Step 5, on every way out of it, ask the person to delete the key in SendGrid and remove its variable from the env file, and note in the plan whether they did. The report reminds them when they didn't. Never print the key, and never write it into another file. + +The steps call this key the full-access provider key. + +## Tracking + +A send follows the account's open and click tracking for each of the two that its `trackingSettings` doesn't set. What PostHog does with opens and clicks is in `mapping.md` under "Open and click tracking". Ask the person once whether open tracking and click tracking are on in SendGrid (Settings, then Tracking), and whether click tracking also covers plain-text emails, and note the answers in the plan. + +A site's emails had no open pixel when its `trackingSettings` sets `openTracking` with `enable: false`, or sets no `openTracking` while the account's open tracking is off. When it sets none and the person can't say whether open tracking is on, nobody can tell whether they had one. Step 3 says under "Opens the provider didn't track" what either means. + +SendGrid didn't track a site's clicks when its `trackingSettings` sets `clickTracking` with `enable: false`, or sets no `clickTracking` while the account's click tracking is off. It didn't track the links in a plain-text part either, unless `clickTracking` sets `enableText: true`, or sets no `clickTracking` while the account's click tracking covers plain-text emails: a site whose plain-text part has a link SendGrid didn't track counts as one whose clicks SendGrid didn't track. When it sets no `clickTracking` and the person can't say whether click tracking is on, or whether it covers plain text for a site whose plain-text part has a link, nobody knows whether SendGrid tracked them. Step 3 says under "A template in PostHog" what either means for a site that moves to a template made in PostHog. + +## Mail settings + +More account settings change every send that doesn't set them, and no send field of PostHog's does the same. In the same question, ask the person whether each one is on in SendGrid: + +- **Subscription tracking** (Settings, then Tracking): SendGrid added an unsubscribe link to each email, or filled the tag that stands for it. When it is on, every site whose `trackingSettings` doesn't set `subscriptionTracking` stays, and the reason is `account subscription tracking on`. +- **Google Analytics** (Settings, then Tracking): SendGrid added `utm_` parameters to every link. When it is on, every site whose `trackingSettings` doesn't set `ganalytics` stays, and the reason is `account google analytics on`. +- **Legacy Email Template** (Settings, then Mail Settings, when the account shows it): SendGrid wrapped every email in that template. When it is on, every site stays, and the reason is `account legacy email template on`. +- **Footer** (Settings, then Mail Settings): SendGrid appended the footer's HTML to the HTML part and its text to the text part. When it is on, ask the person for the footer's HTML and text. Each site that moves in the code form appends them to its `html` and `text`, as SendGrid did, and the plan notes it. A site that moves to a template made in PostHog sends no `html` or `text`: ask the person to put the footer at the end of that template too, and note it in the plan. When the person can't give the footer, every site stays, and the reason is `account footer not copied`. + +When the person can't say whether one of these is on, the sites it would change stay, and the reason is `account mail settings not checked`. Note the answers in the plan. + +## Template content + +A site whose template moves needs the content of its active version: Step 3 checks it, and Step 7 brings it into the code, or the person makes a template in PostHog from it when they ask for one. The API returns the content. + +**Copy from the dashboard.** In SendGrid: Email API, then Dynamic Templates, then the template, then its active version. Ask the person for the template's id, the version's subject, its HTML from the code editor, and its plain text when it has one. + +**Read from the API.** `GET https://api.sendgrid.com/v3/templates/{template_id}` with `Authorization: Bearer $SENDGRID_READ_API_KEY`. The answer holds `id`, `name`, `generation` (`dynamic` or `legacy`), and `versions`, a list of `{ id, active, name, subject, html_content, plain_content }`. Take the version whose `active` is `1`. + +Read the person's copies where they saved them. An API answer goes only into the temporary file Step 3 writes, which is deleted as `template-to-code.md` says. Never write the content into the plan or the report. + +## Suppression list + +SendGrid keeps several suppression lists. It skips an address on any of them on every send. + +**For Step 3's question.** SendGrid's suppression list, in Step 3's question about the sites that rely on it, is Bounces, Spam Reports, Invalid Emails, and Global Unsubscribes together. A person adds to them by hand under Suppressions, and the app with the calls `mapping.md` names under "Calls that change the suppression list", such as an unsubscribe handler that adds a global unsubscribe. A global unsubscribe stopped every send to the address at SendGrid, transactional ones too, unless the send bypassed it, and a moved send reaches it, as "Leave out the other lists" says: name Global Unsubscribes in the question. A site that sets `asm` or `mailSettings` already stays, as `mapping.md` says, so the question leaves out group unsubscribes and the bypass settings. + +Whenever the person adds entries to PostHog's list by hand, in the import or a re-import, such as when the import tool is missing, ask only whether the Address Allow List and Bounce Purge are on, as "For the report" says, and for Bounce Purge's two day counts. Tell them the rule of each one that is on: leave out the addresses and domains on the Allow List, which they read in SendGrid themselves, and the bounces older than the day count for their kind, as "Leave out what SendGrid delivered to anyway" says. A setting they can't say is on gets no rule. + +**Export from the dashboard (preferred).** In SendGrid: Suppressions, then each of Bounces, Spam Reports, and Invalid Emails, then **Export CSV**. Ask the person where they saved each file. Read the header row to find the address column. + +**Read from the API.** With `Authorization: Bearer $SENDGRID_READ_API_KEY`: + +- `GET https://api.sendgrid.com/v3/suppression/bounces` +- `GET https://api.sendgrid.com/v3/suppression/spam_reports` +- `GET https://api.sendgrid.com/v3/suppression/invalid_emails` + +Each answers a list of `{ created, email, reason, status }`. Page with `limit=500` and `offset`, adding 500 each time, until a page has fewer than 500 entries. Wait and retry on `429`. + +**Map each entry.** + +| SendGrid list | PostHog `reason` | +|---|---| +| Bounces | `bounce` | +| Invalid Emails | `bounce` | +| Spam Reports | `complaint` | + +**Leave out what SendGrid delivered to anyway.** Before you take the entries, ask the person what "For the report" asks about the Address Allow List and Bounce Purge, unless they answered it in this run, and, when the Allow List is on, ask for its entries. Apply each one on its own: + +- **Address Allow List** on: leave out every entry of the lists whose address or domain is on it. +- **Bounce Purge** on: leave out each bounce older than its number of days from its `created` time. A `status` that starts with `5` is a hard bounce, and one that starts with `4` is a soft one. + +When you take the entries yourself, a setting the person can't say is on, or whose entries or day counts they can't give, leaves out nothing, and the other one still applies. In a re-import, tell the person each setting's rule and how many entries it left out, that it was not checked, or, for an Allow List whose entries they couldn't give, `Address Allow List entries not given`. + +**Leave out the other lists.** + +- **Blocks** are temporary refusals by a receiving server, not a sign that the address is bad. +- **Global unsubscribes and group unsubscribes** are marketing preferences, and PostHog's opt-outs apply to marketing messages only. At SendGrid a global unsubscribe also stopped transactional mail, unless the send bypassed it. Once the sends move, these addresses get transactional email again. The report gives their count, as "For the report" says. + +## For the report + +The count of global unsubscribes and of group unsubscribes, which the import leaves out. Count the rows of the person's exports (Suppressions, then each of Global Unsubscribes and Group Unsubscribes, then **Export CSV**), or the entries of `GET https://api.sendgrid.com/v3/suppression/unsubscribes`, paged as above, and of `GET https://api.sendgrid.com/v3/asm/suppressions`, with the read-only key. When the person has neither, write `unsubscribes not counted`. + +Ask the person once, whether or not the import runs, whether two settings are on in SendGrid (Settings, then Mail Settings). With the **Address Allow List**, SendGrid delivered to an address on it, or at a domain on it, whatever its lists said. With **Bounce Purge**, SendGrid deleted a bounce once it was older than the days set for hard bounces or for soft bounces. When Bounce Purge is on, ask for its two day counts. Then write: + +- Each setting that is on and its rule, never an address or an Allow List entry: the Allow List leaves out the addresses and domains on it, and Bounce Purge the bounces older than the day count for their kind, where a `status` that starts with `5` is a hard bounce and one that starts with `4` is a soft one. Whoever adds entries to PostHog's list by hand, before cutover or in a re-import, leaves out the same entries. After the import, add under the phase how many entries each one left out. +- When Bounce Purge is on: bounces that SendGrid would have purged later stay suppressed in PostHog until someone removes them. +- Each setting the person can't say is on, and Bounce Purge without its day counts: ` not checked`, because PostHog may suppress addresses that SendGrid still delivered to. After an import in which the person couldn't give the Allow List's entries: `Address Allow List entries not given`, for the same reason. diff --git a/context/skills/migrate-email/references/sendgrid/mapping.md b/context/skills/migrate-email/references/sendgrid/mapping.md new file mode 100644 index 00000000..59397470 --- /dev/null +++ b/context/skills/migrate-email/references/sendgrid/mapping.md @@ -0,0 +1,163 @@ +# SendGrid to PostHog mapping + +This file decides, field by field, whether PostHog can send a SendGrid call site. A site moves only when every field it sets moves. Otherwise it stays on SendGrid, and the reason names the first field or feature that keeps it there. `api.md` has the PostHog side in full, and the rewrite file for the site's language, which Step 7 names, has PostHog's names in that language. + +PostHog fields are written in their Node.js names here, such as `replyTo`. Use the name the rewrite file gives for the site's language, such as `reply_to` in Python. + +## Fields + +| SendGrid | PostHog | A site stays on SendGrid when | +|---|---|---| +| `from` | `from` | Never. `{ email, name }` becomes `'Name '`, with the name in double quotes when it holds a comma, a quote, or another special character, and each `\` and `"` inside those quotes escaped with a `\`, such as `"Jane \"JJ\" Doe" `. It needs a verified sender (Step 4) | +| `to` | `to` | It holds more than one address. A list with one entry moves as that entry. `{ email, name }` moves as the bare address: note in the plan row that the To line loses the name, unless the name is the address itself | +| `subject` | `subject` | Never | +| `text`, `html` | `text`, `html` | Never | +| `content` | `text` for `text/plain`, `html` for `text/html` | It holds another type, or two entries of one type | +| `templateId` with `dynamicTemplateData` | the template's email, brought into the code and rendered with `dynamicTemplateData`, sent as `subject`, `html`, and `text`. A template made in PostHog, by its `templateId` with `templateData`, only when the person asks for one | See "Templates". A legacy template, whose id has no `d-` prefix, always stays | +| `cc`, `bcc` | none | Always. PostHog's send doesn't take it yet | +| `replyTo`, `replyToList` | `replyTo`, one address or a list | An entry carries a name, there are more than 10, or the code builds the value at run time and you cannot tell that it holds bare addresses only | +| `attachments` | none | Always. PostHog's send doesn't take it yet | +| `headers` | none | Always: no PostHog SDK takes headers yet | +| `customArgs` | none | Always: no PostHog SDK takes tags yet | +| `categories` | none | Always: no PostHog SDK takes tags yet | +| No `trackingSettings` | none, as "Open and click tracking" says | Opens and clicks follow the account, as "Open and click tracking" below and `account.md` under "Tracking" say | +| `trackingSettings` with `openTracking` or `clickTracking` | none, as "Open and click tracking" says | Opens and clicks follow "Open and click tracking" below. It stays when the settings do anything else, such as `subscriptionTracking`, `ganalytics`, or an `openTracking` `substitutionTag`: PostHog places its own pixel, so the tag would stay in the email as text | +| `sendAt` / `send_at` | none | Always. PostHog's send doesn't take it yet | +| `batchId` / `batch_id` | none | Always. PostHog's send doesn't take it yet | +| `asm` | none | Always: PostHog has no unsubscribe groups on a transactional send | +| `ipPoolName` / `ip_pool_name` | none | Always. PostHog's send doesn't take it yet | +| `mailSettings` / `mail_settings` | none | Always. Sandbox mode is not a send | +| `personalizations` | the fields of its one entry | It holds more than one entry. One entry moves when each field in it moves | + +**Open and click tracking.** PostHog adds its open pixel to every email it sends, and a send in the code form counts no clicks, as Step 7 says. No field changes this: `captureEngagement` decides only whether the email's events land on the recipient's person, so tracking never maps to it. SendGrid decides opens and clicks one at a time. When the send's `trackingSettings` sets `openTracking` or `clickTracking`, that setting's `enable` decides for it. Otherwise the account's setting decides, as `account.md` says under "Tracking". + +- Opens: `account.md` says under "Tracking" which sites' emails had no open pixel, and for which nobody can tell. Those sites follow Step 3 under "Opens the provider didn't track". +- Clicks: when SendGrid tracked a moved site's clicks, or nobody knows whether it did, note in the plan row that PostHog counts no clicks for it, unless it moves to a template made in PostHog that is not marked sensitive. When SendGrid didn't track them, or nobody knows, as `account.md` says under "Tracking", a site that moves to a template made in PostHog follows Step 3 under "A template in PostHog". + +**Account mail settings.** SendGrid's subscription tracking, Google Analytics, Legacy Email Template, and Footer settings change a send that sets no field for them. `account.md` says under "Mail settings" which sites they keep on SendGrid, and how a moved site keeps the footer. + +## Calls + +| SendGrid call | A site stays on SendGrid when | +|---|---| +| `sgMail.send(msg)`, `SendGridAPIClient.send`, `sg.client.mail._('send').post`, `$sendgrid->send`, `client.Send`, `SendEmailAsync`, `POST /v3/mail/send` | A field above keeps it | +| `sgMail.send(msg, true)`, `sgMail.sendMultiple`, `sgMail.send([...])`, `Mail(..., is_multiple=True)`, `CreateSingleEmailToMultipleRecipients`, `CreateMultipleEmailsToMultipleRecipients` | Always: each sends several emails in one call | +| SMTP through `smtp.sendgrid.net` | Always. PostHog has no SMTP relay | +| A framework mailer (Django, Rails, Laravel, Swoosh, Bamboo) | Always. PostHog has no mailer driver | +| A send in a language without a PostHog SDK that sends email, such as Java or Kotlin | Always. Step 7 lists the languages PostHog's SDKs cover | +| A send through an `@sendgrid/mail` client that sets `sgMail.setSecretRules(...)` | Always. That client refuses a send whose content matches a rule before any request, and PostHog's send has no such check | + +## Code around the send + +SendGrid's SDKs report an error in two ways. The Node.js and Python SDKs throw. The Ruby, PHP, Go, and .NET SDKs answer a response, and the code reads its status code. The rewrite file says under "The answer" how PostHog's SDK reports an error in its language. Keep the site failing the same way it did, as the Rewrite section says. + +| Code | A site stays on SendGrid when | +|---|---| +| A `catch` or `except` that reads `err.code`, `err.response.body.errors`, `e.status_code`, or `e.body` | It branches on a SendGrid error text or `field` value. A branch on the HTTP status moves: read PostHog's status code | +| A check of `response.statusCode`, `status_code`, `statusCode()`, or `StatusCode` | Never. Read PostHog's error instead, as the Rewrite section says | +| The `X-Message-Id` header stored for a webhook or a status read | Always, when the Event Webhook or the Email Activity API uses it | +| The `X-Message-Id` header logged or stored for the app's own use | Never. Use PostHog's message id. When the code after the send can't go on without an id, such as a column that must hold one, the `recipient_suppressed` branch below has none to give: the site stays, and the reason is `needs a message id for a suppressed send`, unless that branch can skip the code that needs it | +| A send the code can cancel: a request with a `signal` a caller passes in or an `AbortController` it aborts, a Go context a caller can cancel given to `SendWithContext` or `MakeRequestWithContext`, or a .NET `CancellationToken` a caller can cancel given to `SendEmailAsync` | Unless the PostHog send in that language takes the same cancellation, as Go's `Send(ctx, ...)` and .NET's `SendAsync(..., cancellationToken)` do: pass it there. Otherwise always, as in Node.js, whose send takes no signal: an email the old code cancelled would go out. A fixed deadline moves, as "HTTP" under "Rewrite" says | + +## Templates + +A dynamic template site moves by bringing the template into the code, as `template-to-code.md` says. The check needs the content of the template's active version, because the code has only its id. `account.md` says how to get it. The code renders the version's HTML, plain text, and subject with the site's `dynamicTemplateData`, and the send sets the rendered `subject`, `html`, and `text`. + +SendGrid templates are Handlebars. In the code: + +| SendGrid | In the code | +|---|---| +| `{{name}}`, `{{ name }}` | The value, escaped. A missing value renders as nothing | +| `{{user.first_name}}` | The nested value, escaped. A missing step of the path renders as nothing | +| `{{{name}}}` | The value as it is, unescaped, under the rule of `template-to-code.md`. In the subject and the plain text, it is the plain value | +| `{{insert name "default=there"}}`, `{{insert name "there"}}` | The value, escaped, or `there` when the value is missing. When the code can pass `null` or `""` for it, use the fallback for those too, and note it in the plan row: SendGrid's docs don't say what `insert` does with them | +| `{{#if a}}…{{else if b}}…{{else}}…{{/if}}` | A branch that tests each value for Handlebars truth, in order | +| `{{#unless a}}…{{else}}…{{/unless}}` | The same branch, with the test turned around | +| `{{#each items}}…{{else}}…{{/each}}` | A loop over the list, or over the values of an object in key order. The `{{else}}` part shows when there is nothing to walk | +| Inside `{{#each}}`: `{{this}}`, `{{this.name}}`, or a bare `{{name}}` | The current item, or its field. A bare name reads the current item only: when the item has no such field, it renders as nothing, even when a value outside the loop has that name. Only `../name` and `@root.name` read outside the loop | +| Inside `{{#each}}`: `{{@index}}`, `{{@key}}`, `{{@first}}`, `{{@last}}`, `../name`, `@root.name` | The item's position from 0, its key in an object, whether it is the first or the last, and the value one level out or at the top | +| The version's `subject` | The send's `subject`, rendered with the same conversions. SendGrid escapes HTML in a subject too, and a mail client shows the entities as they are, so the subject needs Handlebars' own escape, character for character: `&` as `&`, `<` as `<`, `>` as `>`, `"` as `"`, `'` as `'`, `` ` `` as ```, and `=` as `=`. Write it as a small function next to the email, never the HTML escape of `template-to-code.md`, which writes some of these differently. So `{{name}}` with `A&B` stays `A&B` as SendGrid sent it, and only `{{{name}}}` is the plain value. This is SendGrid's rule, so it takes the place of `template-to-code.md`'s rule never to escape a subject. Note each such value in the plan row. When the version has no subject, the send takes the site's own `subject` as it is. A site whose version has no subject and that sets none stays: PostHog's send needs a subject | +| The version's plain text | The send's `text`, rendered with the same conversions. SendGrid escapes HTML in the plain text too, so the text takes the subject's escape, from the same function: `{{name}}` with `A&B` stays `A&B` as SendGrid sent it, and only `{{{name}}}` is the plain value. This takes the place of `template-to-code.md`'s rule never to escape the text part. Note each such value in the plan row. A version without plain text sends `html` only | + +- Handlebars truth: `false`, a missing value, `null`, `""`, `0`, and an empty list are false. Everything else is true, including `"0"`, `"false"`, and an empty object. Write that test out in the code, as `template-to-code.md` says. +- Any other helper, a partial (`{{> name}}`), or an `{{unsubscribe}}`, `{{{unsubscribe}}}`, `{{weblink}}`, or `{{{unsubscribe_preferences}}}` tag that SendGrid fills from the account keeps the template on SendGrid. Such tags come from unsubscribe groups, which a transactional send through PostHog doesn't have. +- A SendGrid template holds no sender, so the send takes the site's own `from` and `replyTo`. A `from` or `replyTo` that holds a `{{` keeps the site on SendGrid, with the reason `placeholder in from or reply-to`: these files don't say whether SendGrid fills it from `dynamicTemplateData`. A site that sets `subject` next to `templateId` moves only when the version has no subject, or when the site's subject is the same in every send as the version's: which one SendGrid sent can't be told from the code. Otherwise it stays. +- Images that the template's editor uploaded to SendGrid have URLs on `marketing-image-production.s3.amazonaws.com` or `mc.sendgrid.com`. Keep them, and list them, as `template-to-code.md` says. + +### Templates made in PostHog + +Only for a template the person asks to make in PostHog, as Step 3 says. The person makes it from the SendGrid template and gives you its template ID, and makes it transactional once a verified sender covers its address, as Step 3 says. Until they give the ID, the site stays on SendGrid. + +An email that carries a login code, a reset link, or another secret needs its PostHog template marked sensitive: PostHog then keeps no copy of each sent email's body or filled-in subject, and sends its links as they are, without its click redirect. Templates are not sensitive by default. + +SendGrid's Handlebars becomes Liquid in the PostHog template. Values move under `variables`, and each name keeps its case: + +| SendGrid | PostHog | +|---|---| +| `{{name}}` | `{{ variables.name }}` | +| `{{user.first_name}}` | `{{ variables.user.first_name }}` | +| `{{#if paid}}…{{else}}…{{/if}}`, when `paid` is a boolean | `{% if variables.paid %}…{% else %}…{% endif %}` | +| `{{#unless paid}}…{{/unless}}`, when `paid` is a boolean | `{% unless variables.paid %}…{% endunless %}` | +| `{{#each items}}{{this.name}}{{/each}}` or `{{#each items}}{{name}}{{/each}}`, when `items` is a list | `{% for item in variables.items %}{{ item.name }}{% endfor %}` | +| `{{this}}` inside `{{#each items}}` | `{{ item }}` | +| `{{insert name "default=there"}}` | `{{ variables.name \| default: 'there' }}` | +| The version's `subject` | The PostHog template's subject, with the same conversions | + +Write the `\|` above as a plain `|`. The table escapes it. + +- Handlebars `{{name}}` escapes HTML, and so does PostHog's Liquid in the HTML part, so plain values render the same there. The subject and the plain text differ: SendGrid escaped their values too, as the subject and plain text rows of the table for the code say, and PostHog inserts their values as they are. A template whose subject or plain text has a `{{name}}` or `{{insert}}` value that can hold `&`, `<`, `>`, `"`, `'`, `` ` ``, or `=` stays on SendGrid, and the reason is `escaped value in subject or plain text`. Read what the site sends for each. +- Handlebars treats `0`, an empty string, and an empty list as false, and Liquid treats them as true. Read what the site sends for each `if` and `unless` value. A value that is not always a boolean, `null`, or missing keeps the template on SendGrid. +- Handlebars `{{#each}}` also walks the values of an object, and Liquid's `for` doesn't. An `each` over a value that is not always a list keeps the template on SendGrid. +- Inside `{{#each}}`, a bare `{{name}}` reads the current item, as `{{this.name}}` does. It becomes `{{ item.name }}`, never `{{ variables.name }}`. +- `{{{name}}}` inserts HTML as it is. PostHog's Liquid escapes it, so a template that uses triple braces stays on SendGrid. +- Any other helper, such as `{{#equals}}`, `{{formatDate}}`, `{{#greaterThan}}`, `{{length}}`, or `{{#and}}`, keeps the template on SendGrid. So does `{{@index}}`, a partial, and a block inside `{{#each}}` that reads a value outside the loop with `../`. +- Liquid's `default` also replaces an empty string. When the code can pass an empty string for such a value, the email shows the fallback. Note it in the report. +- The PostHog template carries its own sender, reply-to address, and subject. A site that sets `from`, `replyTo`, or `subject` next to `templateId` moves only when the value is the same in every send, and the person puts it into the PostHog template. Otherwise it stays. + +## Webhooks + +The Event Webhook stops getting events for a site that moves. PostHog records the events of a moved send on the person it names with `distinctId`, when engagement capture is on: + +| SendGrid event | PostHog event | +|---|---| +| `processed` | `$workflows_email_sent` | +| `delivered` | `$workflows_email_delivered` | +| `bounce` | `$workflows_email_bounced` | +| `spamreport` | `$workflows_email_blocked` | +| `open` | `$workflows_email_opened` | +| `click` | `$workflows_email_link_clicked`, only for a template made in PostHog that is not marked sensitive. A send in the code form counts no clicks | +| `dropped` | none. PostHog refuses the send instead, such as with `recipient_suppressed` | +| `deferred`, `unsubscribe`, `group_unsubscribe`, `group_resubscribe` | none | + +PostHog's events carry no SendGrid `sg_message_id` and no `custom_args`. A site whose events a handler matches by message id or by a custom arg stays on SendGrid. For every other handler, the report tells the person which events it handles, which PostHog events take their place, and that a webhook destination in PostHog (Data pipelines, then Destinations) can post them to an endpoint. The handler itself keeps working for the sites that stay. Never change it. + +A site whose emails a handler acts on for an event that has no PostHog event in the table, such as `dropped` or `deferred`, stays, and the reason is `its events feed a webhook handler`. SendGrid sends `unsubscribe`, `group_unsubscribe`, and `group_resubscribe` only for emails with subscription tracking or `asm`, which keep their sites on SendGrid already, so a handler on those events keeps no other site. A site whose emails a `click` handler acts on moves only to a template made in PostHog that is not marked sensitive. Otherwise it stays, and the reason is `its clicks feed a webhook handler`. + +## Rewrite + +How a site that moves is written with PostHog's SDK. Step 7 says when to use each part, and the rewrite file for the site's language has the client and the send. + +**The PostHog client.** Set `migratedFrom: 'sendgrid'` on the client that Step 7 configures, when its client options take it. + +**The call.** The SendGrid send becomes one PostHog send. Build its fields from the message the site built. + +**The fields.** + +- `to` becomes a bare address. A list with one entry becomes that entry. When a helper takes a list of recipients and every caller passes one address, change the parameter to one address and update the callers. Never send one email per address in a loop. + +**HTTP.** A request to `POST /v3/mail/send`, through `fetch`, axios, `requests`, `@sendgrid/client`, or another HTTP client, becomes the SDK's send in that language, with the fields "The call" names. A site's own deadline is a timeout the code sets on that request, or on the SendGrid client it sends through: `signal: AbortSignal.timeout(5000)` or `requests.post(..., timeout=5)` on a request; `CURLOPT_TIMEOUT` in the PHP client's `curl` options; the `Timeout` of the `HttpClient` given to the .NET client, or a token that times out given to `SendEmailAsync(msg, token)`; and the `Timeout` of the Go client's `HTTPClient`, or a context with a deadline given to `SendWithContext` or `MakeRequestWithContext`. The site keeps that deadline, as Step 7 says under "Set up the client". A timeout the code doesn't set, such as a SendGrid client's own default, is none. When a site with a deadline also retried, note in the plan row that its send no longer retries. + +**Templates made in PostHog.** The site sends `templateId` with the template ID the person gave, and `templateData` with the site's `dynamicTemplateData`. It sends no `templateKey`, `subject`, `html`, `text`, `from`, or `replyTo`: the template holds them. + +**Errors.** Keep the site failing the same way it did: + +- **Node.js.** `posthog.emails.send` never throws on an API error: it answers `{ data, error }`. Where `sgMail.send` threw into a `catch`, throw from the error so the same `catch` runs: `if (error) throw Object.assign(new Error(error.message), { code: error.statusCode ?? error.code, name: error.name, posthogCode: error.code })`. `err.code` keeps holding the HTTP status, and holds `network_error` when the request never reached PostHog, `err.posthogCode` holds PostHog's own code, so a branch on `err.code === 429` also checks `err.posthogCode !== 'quota_limited'`, as Step 7 says, and a check of `err.code === 400` becomes a check of `err.name === 'validation_error'`. A read of `err.response.body` becomes a read of `err.message`. PostHog's `data` is an object, not SendGrid's `[response, body]` list: never destructure it as a list, and rewrite each read of the answer, such as `const [response] = await sgMail.send(msg)` or `result[0]`, as "The answer" below says. +- **Python.** PostHog's SDK raises too. When the same `try` also holds a SendGrid call that stays, such as a suppression call, keep each `except HTTPError` handler for that call's errors, and add the PostHog exception's handling next to it. +- **Ruby, PHP, Go, and .NET.** The code read SendGrid's status code. Read PostHog's error the way the rewrite file says under "The answer", and keep each branch: a status of 400 or more becomes "there is an error", a branch on `400` reads PostHog's error name `validation_error`, and a branch on another status, such as 429, reads PostHog's status code. Their SendGrid clients don't raise on a refused send: a site that never read the status went on after one, and only a network failure raised (Ruby, PHP, .NET) or set `err` (Go). A .NET client whose options set `HttpErrorAsException` is the exception: it threw on a refused send, so its sites throw PostHog's exception the way they threw before, and the next sentence does not apply to them. Such a site logs PostHog's error and goes on, and raises or returns it when its code is `network_error`, as Step 7 says. A Go site that sent through `MakeRequestRetry` or `MakeRequestRetryWithContext` also got `err` when SendGrid still answered 429 after its last retry (`rate limit retry exceeded`): it also returns PostHog's error when its status code is 429, which PostHog's send answers only once its own retries are used up. A .NET site whose SendGrid client retries server errors stays, and the reason is `its client retries server errors`: the client retried a server error, SendGrid's `RetryDelegatingHandler` threw `HttpRequestException` after its last retry, and PostHog's send does neither. The client retries when its options set `ReliabilitySettings` with `MaximumNumberOfRetries` above 0 and it makes its own `HttpClient`, or when the code adds a handler that retries to the `HttpClient` the client gets, the one it passes to the client or the one of the client `AddSendGrid` registers: `RetryDelegatingHandler` with `MaximumNumberOfRetries` above 0, a Polly policy, or a resilience handler such as `AddStandardResilienceHandler`. `ConfigureHttpClientDefaults` adds its handlers to every client from .NET's factory, and often sits in another project, such as an Aspire app's service defaults: search the whole solution for it before a site whose `HttpClient` comes from .NET's factory moves, an `AddSendGrid` site or one given `CreateClient`'s client alike. `AddSendGrid` gives the client an `HttpClient` from .NET's factory, so the `ReliabilitySettings` in its options retry nothing by themselves. When the code can't tell what a handler it adds there does, or how many times it retries, the site stays too. +- PostHog refuses a send to a suppressed address with `recipient_suppressed`, where SendGrid took it and dropped it later. When the site throws, retries, or reports a failure on any error, handle `recipient_suppressed` before that branch the way the site handled a send that went out: log it and go on. Note each site that got this branch in the report. + +**The answer.** The message id of a moved send is PostHog's. Code that logged `X-Message-Id`, or stored it for the app's own use, takes PostHog's message id instead. + +**Calls that change the suppression list.** Step 3 and Step 5 say what to do with each of these calls. They are not sends, and they stay on SendGrid: a request, through a SendGrid client or a raw HTTP call, to `/v3/suppression/bounces`, `/v3/suppression/invalid_emails`, `/v3/suppression/spam_reports`, `/v3/suppression/blocks`, or `/v3/asm/suppressions/global`, or to an unsubscribe group's suppressions under `/v3/asm/groups/{group_id}/suppressions`, such as a `DELETE` request to `/v3/suppression/bounces/{email}`. + +**The SendGrid client.** Keep the SendGrid client, its import, and `setApiKey` while any site or other call still uses them. When nothing uses them, remove the import and the client, and keep the package and its key: the person removes them after the cutover. diff --git a/context/skills/migrate-email/references/sendgrid/sdk-reference.md b/context/skills/migrate-email/references/sendgrid/sdk-reference.md new file mode 100644 index 00000000..79ed554d --- /dev/null +++ b/context/skills/migrate-email/references/sendgrid/sdk-reference.md @@ -0,0 +1,182 @@ +# SendGrid SDK reference + +This file says what SendGrid looks like in a codebase. It does not describe PostHog. `mapping.md` maps each field to PostHog. + +## Key + +The API key lives in `SENDGRID_API_KEY` by convention. Apps also use their own names, such as `SENDGRID_KEY`: take the name the client reads. + +## Packages + +| Language | Package | Send | +|---|---|---| +| Node.js | `@sendgrid/mail`, and `@sendgrid/client` for raw API requests | `sgMail.send(msg)` | +| Python | `sendgrid` (PyPI) | `SendGridAPIClient(key).send(Mail(...))` | +| Ruby | `sendgrid-ruby` | `SendGrid::API.new(api_key:).client.mail._('send').post(request_body:)` | +| PHP | `sendgrid/sendgrid` | `(new \SendGrid($key))->send($email)` | +| Go | `github.com/sendgrid/sendgrid-go` with `helpers/mail` | `sendgrid.NewSendClient(key).Send(message)` | +| .NET | `SendGrid` (NuGet), and `SendGrid.Extensions.DependencyInjection` | `client.SendEmailAsync(msg)` | +| Java | `com.sendgrid:sendgrid-java` | `sg.api(request)` with `Mail` | + +These are not SendGrid's send SDK, and alone they are not a migration: `@sendgrid/eventwebhook` checks webhook signatures, `@sendgrid/inbound-mail-parser` reads inbound mail, and `nodemailer` sends through any SMTP server. + +## Clients and sends + +**Node.js.** `sgMail.send(msg)` answers `[response, body]`: `response.statusCode` is `202`, and `response.headers['x-message-id']` is the message id. It throws a `ResponseError` on an API error: `err.code` is the HTTP status, and `err.response.body.errors` is a list of `{ message, field, help }`. + +```js +import sgMail from '@sendgrid/mail' +sgMail.setApiKey(process.env.SENDGRID_API_KEY) + +await sgMail.send({ + to: 'jane@example.com', + from: { email: 'hello@mail.example.com', name: 'Example' }, + subject: 'Welcome', + text: 'Welcome', + html: '

Welcome

', +}) +``` + +- `sgMail.send(msg)` with a list in `to` sends **one** email that every address sees. +- `@sendgrid/client`: `client.request({ method: 'POST', url: '/v3/mail/send', body })` sends the HTTP body below. + +**Python.** It raises `python_http_client.exceptions.HTTPError` and its subclasses, such as `BadRequestsError` and `TooManyRequestsError`, on an API error. `e.status_code` and `e.body` hold the answer. A success answers a response with `status_code` `202` and the message id in `headers['X-Message-Id']`. + +```python +from sendgrid import SendGridAPIClient +from sendgrid.helpers.mail import Mail + +message = Mail( + from_email=("hello@mail.example.com", "Example"), + to_emails="jane@example.com", + subject="Welcome", + html_content="

Welcome

", + plain_text_content="Welcome", +) +response = SendGridAPIClient(os.environ["SENDGRID_API_KEY"]).send(message) +``` + +- `message.template_id` and `message.dynamic_template_data` send a dynamic template. +- `message.add_cc(...)`, `message.add_bcc(...)`, `message.reply_to = ReplyTo(...)`, `message.attachment = Attachment(FileContent(...), FileName(...), FileType(...), Disposition(...), ContentId(...))`, `message.category = Category(...)`, `message.custom_arg = CustomArg(...)`, `message.header = Header(...)`, `message.send_at = SendAt(...)`, and `message.asm = Asm(...)` set the other fields. +- `sg.client.mail.send.post(request_body=data)` sends a raw HTTP body. + +**Ruby.** The send answers a response with `status_code` (a string, such as `"202"`), `body`, and `headers`. It does not raise on an API error: the code reads `status_code`. + +```ruby +require 'sendgrid-ruby' + +mail = SendGrid::Mail.new( + SendGrid::Email.new(email: 'hello@mail.example.com', name: 'Example'), + 'Welcome', + SendGrid::Email.new(email: 'jane@example.com'), + SendGrid::Content.new(type: 'text/html', value: '

Welcome

') +) +sg = SendGrid::API.new(api_key: ENV['SENDGRID_API_KEY']) +response = sg.client.mail._('send').post(request_body: mail.to_json) +``` + +A `SendGrid::Personalization` with `add_to`, `add_cc`, `add_bcc`, and `add_dynamic_template_data` builds the recipients, and `mail.template_id`, `mail.reply_to`, `mail.add_attachment`, `mail.add_category`, `mail.add_custom_arg`, `mail.add_header`, `mail.send_at`, and `mail.asm` set the rest. + +**PHP.** `send` answers a response with `statusCode()`, `headers()`, and `body()`. It does not throw on an API error, only on a network error. + +```php +$email = new \SendGrid\Mail\Mail(); +$email->setFrom('hello@mail.example.com', 'Example'); +$email->setSubject('Welcome'); +$email->addTo('jane@example.com'); +$email->addContent('text/html', '

Welcome

'); +$response = (new \SendGrid(getenv('SENDGRID_API_KEY')))->send($email); +``` + +The other setters are `addCc`, `addBcc`, `setReplyTo`, `addAttachment($content, $type, $filename, $disposition, $contentId)`, `setTemplateId`, `addDynamicTemplateDatas`, `addCategory`, `addCustomArg`, `addHeader`, `setSendAt`, and `setAsm`. + +**Go.** `Send` answers `(*rest.Response, error)`. `err` is set only for a network error: an API error is a `response.StatusCode` of 400 or more. The message id is in `response.Headers["X-Message-Id"]`. + +```go +from := mail.NewEmail("Example", "hello@mail.example.com") +to := mail.NewEmail("", "jane@example.com") +message := mail.NewSingleEmail(from, "Welcome", to, "Welcome", "

Welcome

") +response, err := sendgrid.NewSendClient(os.Getenv("SENDGRID_API_KEY")).Send(message) +``` + +`mail.NewV3Mail()` with `mail.NewPersonalization()` (`AddTos`, `AddCCs`, `AddBCCs`, `SetDynamicTemplateData`) builds the full form, with `SetTemplateID`, `SetReplyTo`, `AddAttachment`, `AddCategories`, `SetCustomArg`, `SetHeader`, `SetSendAt`, and `SetASM`. + +**.NET.** `SendEmailAsync` answers a `Response` with `StatusCode`, `IsSuccessStatusCode`, `Headers`, and `Body`. It does not throw on an API error unless the client options set `HttpErrorAsException`. + +```csharp +var client = new SendGridClient(Environment.GetEnvironmentVariable("SENDGRID_API_KEY")); +var msg = MailHelper.CreateSingleEmail( + new EmailAddress("hello@mail.example.com", "Example"), + new EmailAddress("jane@example.com"), + "Welcome", "Welcome", "

Welcome

"); +var response = await client.SendEmailAsync(msg); +``` + +`MailHelper.CreateSingleEmailToMultipleRecipients` and `CreateMultipleEmailsToMultipleRecipients` send to several addresses. A `SendGridMessage` takes `SetFrom`, `AddTo`, `AddCc`, `AddBcc`, `SetReplyTo`, `AddAttachment(filename, base64Content, type, disposition, contentId)`, `SetTemplateId`, `SetTemplateData`, `AddCategory`, `AddCustomArg`, `AddHeader`, `SetSendAt`, and `SetAsm`. Apps that use dependency injection take an `ISendGridClient`. + +**HTTP.** `POST https://api.sendgrid.com/v3/mail/send`, or `api.eu.sendgrid.com` for an EU subuser, with `Authorization: Bearer `. A success answers `202` with no body and the message id in the `X-Message-Id` header. An error answers `{ "errors": [{ "message", "field", "help" }] }`. Look for `api.sendgrid.com` in `fetch`, `axios`, `got`, `requests`, `httpx`, `net/http`, and `curl` calls. + +**EU subusers.** An app on an EU subuser either names `api.eu.sendgrid.com`, or sets the client's data residency to `eu`, such as `client.setDataResidency('eu')` on `@sendgrid/client` in Node.js or `sg.set_sendgrid_data_residency("eu")` in Python. + +```json +{ + "personalizations": [{ "to": [{ "email": "jane@example.com" }], "dynamic_template_data": { "name": "Jane" } }], + "from": { "email": "hello@mail.example.com", "name": "Example" }, + "subject": "Welcome", + "content": [{ "type": "text/plain", "value": "Welcome" }, { "type": "text/html", "value": "

Welcome

" }] +} +``` + +**SMTP.** SendGrid takes mail over SMTP at `smtp.sendgrid.net`, with the user `apikey` and the API key as the password. Look for that host in Nodemailer transports, `EMAIL_HOST`, `MAIL_HOST`, or `SMTP_HOST` settings, and framework mail config. An `X-SMTPAPI` header carries SendGrid options over SMTP. + +**Frameworks.** Mailers that send every framework mail through SendGrid: `django-sendgrid-v5` and `django-anymail` with its SendGrid backend in Django, `sendgrid-actionmailer` in Rails, `s-ichikawa/laravel-sendgrid-driver` (`MAIL_MAILER=sendgrid`) in Laravel, and the SendGrid adapters of Swoosh and Bamboo in Elixir. Each mailable or mailer that sends is a site. + +## Send fields + +Node.js names, with the HTTP names where they differ. The other SDKs use the same names in their own case. + +| Node.js field | HTTP field | Value | +|---|---|---| +| `from` | `from` | An address, or `{ email, name }` | +| `to` | `personalizations[].to` | An address, `{ email, name }`, or a list of them | +| `cc`, `bcc` | `personalizations[].cc`, `bcc` | Same as `to` | +| `replyTo` | `reply_to` | An address, or `{ email, name }` | +| `replyToList` | `reply_to_list` | A list of `{ email, name }` | +| `subject` | `subject` | String | +| `text`, `html` | `content` | Strings. HTTP sends `[{ type: 'text/plain', value }, { type: 'text/html', value }]` | +| `content` | `content` | A list of `{ type, value }` | +| `templateId` | `template_id` | A dynamic template (`d-...`), or a legacy template | +| `dynamicTemplateData` | `personalizations[].dynamic_template_data` | Object that fills a dynamic template | +| `substitutions` | `personalizations[].substitutions` | Values for tags in the subject, the content, and the reply-to address, such as `-name-` | +| `attachments` | `attachments` | A list of `{ content, filename, type, disposition, contentId }`. `content` is base64 | +| `categories` | `categories` | A list of strings | +| `customArgs` | `custom_args` | Object of string values, sent back in webhook events | +| `headers` | `headers` | Object of custom headers | +| `sendAt` | `send_at` | A Unix time, up to 72 hours ahead | +| `batchId` | `batch_id` | A batch that can be cancelled or paused | +| `asm` | `asm` | `{ groupId, groupsToDisplay }`: an unsubscribe group | +| `ipPoolName` | `ip_pool_name` | String | +| `mailSettings` | `mail_settings` | Such as `sandboxMode`, `bypassListManagement`, `footer` | +| `trackingSettings` | `tracking_settings` | Such as `clickTracking`, `openTracking`, `subscriptionTracking`, `ganalytics` | +| `personalizations` | `personalizations` | A list of recipient groups, each with its own `to`, `cc`, `bcc`, `subject`, data, and `sendAt` | + +## Templates stored at SendGrid + +A send with `templateId: 'd-...'` and `dynamicTemplateData` renders a dynamic template from the SendGrid dashboard. Its content is not in the code. + +- A template has versions. The active version holds `subject`, `html_content`, and `plain_content`. +- A legacy template has an id without the `d-` prefix, and reads `substitutions` as `-name-` tags. + +## Status reads + +The Email Activity API (`GET https://api.sendgrid.com/v3/messages`) reads a message's delivery state. Code that reads it reads SendGrid's state. + +## Webhooks + +The Event Webhook posts a JSON list of events to an endpoint of the app. Each event has `email`, `timestamp`, `event`, `sg_event_id`, `sg_message_id`, and the send's `custom_args` and `category`. `event` is one of `processed`, `dropped`, `deferred`, `delivered`, `bounce`, `open`, `click`, `spamreport`, `unsubscribe`, `group_unsubscribe`, and `group_resubscribe`. + +Look for routes that read `sg_message_id` or `sg_event_id`, `EventWebhook` and `verifySignature` from `@sendgrid/eventwebhook`, and the `X-Twilio-Email-Event-Webhook-Signature` header. + +## Other APIs + +Marketing Campaigns, contacts, lists, segments, unsubscribe groups, suppressions, templates, API keys, subusers, and stats (`/v3/marketing/*`, `/v3/contactdb/*`, `/v3/asm/*`, `/v3/suppression/*`, `/v3/templates`, `/v3/api_keys`, `/v3/stats`) are not transactional sends. diff --git a/scripts/lib/tests/migrate-email-skill.test.js b/scripts/lib/tests/migrate-email-skill.test.js index 0cff7a9c..d94b7980 100644 --- a/scripts/lib/tests/migrate-email-skill.test.js +++ b/scripts/lib/tests/migrate-email-skill.test.js @@ -11,7 +11,7 @@ const DOCS_PAGES = { 'https://posthog.com/docs/workflows/transactional-emails/api.md': '# Send transactional emails with the API\n', }; const REFERENCE_PATH = /references\/[A-Za-z0-9_./-]+\.md/g; -const PROVIDERS = ['resend']; +const PROVIDERS = ['resend', 'sendgrid']; const PROVIDER_FILES = ['sdk-reference.md', 'mapping.md', 'account.md']; const SDKS = ['node', 'python', 'go', 'ruby', 'php', 'dotnet', 'rust', 'elixir'];