Skip to content

feat(analytics): report detected email providers and identify signup runs - #1506

Merged
Silthus merged 15 commits into
mainfrom
feat/detected-email-providers
Nov 13, 2026
Merged

Silthus merged 15 commits into
mainfrom
feat/detected-email-providers

Conversation

@Silthus

@Silthus Silthus commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The Wizard's codebase scan already finds the email services a user's code sends through, such as Resend, SendGrid or Amazon SES. That fact stays in the scan, so we can't count how many Wizard users already send email from code, or follow them into Workflows.

With this PR, a run that finds an email provider sends one wizard: email providers detected event with email_provider_kinds (the detected kinds, such as ["Resend"]). Nothing is set on the person. This covers every program that scans: the default integration flow, interactive or CI, wizard warehouse and wizard self-driving. A wizard --signup run now identifies the account it creates, so its events land on that account's person too.

Impact

Pays into Grow Workflows through Started orgs (orgs that create their first workflow). Not measured yet: an insight in the internal project counts people who sent wizard: email providers detected per week, and how many of them reach Workflows, start its onboarding, create an email sender, or launch a workflow or broadcast. In scope: email, the Messaging category. Expected move: small, up. On its own this PR only adds the signal. A feature flag can't target an event, so a follow-up that targets these users, such as a "We found Resend in your code" card in Workflows, needs a cohort or its own check. Run cost: n/a.

Merge Danger

Caution

🔴 Merge risk: high · one-way door once the next Wizard release: the person writes and alias merges of signup runs stay

  • wizard --signup runs, in both regions, now read the new account's user and identify it: $identify sets that account's own email and name on its person, and $create_alias merges the run's pre-login id into it, as OAuth and API-key logins already do through logIn. Those writes stay once sent. A failed or slow read leaves the signup going unidentified after at most 10 seconds, with one exception. Covered by the shadow diff (read answers, 403, after 12 seconds, never) and signup.test.ts. Uncovered row: a read that fails on an untrusted certificate chain just before the deadline still runs the chain check main runs on every user read, with its own 5-second limit, so that signup goes on up to 5 seconds later.
  • Signup runs that read their user also get what every login with a user gets: the run screen greets them by first name (Welcome, New. where it said Welcome.), their later events carry the new account's organization group, and the Wizard's feature flags are evaluated as that user. Every flag resolves as before for a new account: the Wizard's own flag evaluation against live PostHog returned the same value for all eight flags, anonymous and identified. The AI opt-in gate still skips signup runs. Covered by the shadow diff, including the real learn card's first frame.
  • Runs that log in with a known user and whose scan finds an email provider send one new wizard: email providers detected event with the provider kinds. Nothing is written to the person. Covered by the shadow diff, the live run (a signup run only) and email-providers.test.ts, which checks every person write but seeds the logged-in user instead of identifying the analytics client. The post-deploy watch then confirmed in production that the event lands on identified runs after the other logins too.
  • Every other user read (OAuth and API-key logins, the AI opt-in screen) and the project read are unchanged: the shadow diff shows the same outcome as main for a user read that answers at once, after 12 seconds, or never, and for user and project reads of a remote host that recover on a retry or run out of retries, with the same requests and timing.
  • Switch-off: none: live in the next Wizard release, fix forward.

Rollout

  • Stages: merge; release-please adds it to the next release PR; publishing that release ships it to every run of the latest Wizard, both regions.
  • Decided: signup identify mirrors the login identify, and only the signup's user read gets a deadline (Decisions below).
  • Watch: signup runs' $exception from fetchUserData and their wizard: auth complete in the Wizard's analytics project, for a week after the release. No alert covers either.
  • Tier: critical (b: high; c: signup identify writes stay once released)

Changes

  • Which providers: detectedEmailProviders (src/services/codebase-scan/email-providers.ts) reads the run's codebase scan and keeps the tool catalog kinds of email services: AwsSes, Beehiiv, BigMailer, Brevo, CampaignMonitor, ConvertKit, Drip, Elasticemail, EmailOctopus, Flexmail, Loops, Mailchimp, MailerLite, MailerSend, Mailgun, Mailjet, Plunk, Postmark, Resend, SendGrid, Shipmail, Smaily, SparkPost, Tinyemail. Kind names match PostHog's live /api/wizard/tool_catalog/. The kinds are a typed list, EmailProviderKind.
  • When it goes out: reportCodebaseScan, the one place a scan is cleared to leave the machine, captures wizard: email providers detected when it shares the scan: sharing granted, telemetry on, logged in with a known user, and the scan did not fail. A login without a user (a signup whose user read failed, or an API key without user:read) sends nothing. The scan-share latch keeps it to once per run. A scan with no email provider sends nothing. The event carries only the provider kind names, from the fixed list above.
  • Signup runs: askForProvisioningSignup (src/tui/auth/project-data.ts) reads the new account's user with the provisioned token, which already holds user:read, identifies it, and returns it as the login's user. Before, a signup run never identified anyone, so every event stayed on an anonymous person. A failed read leaves the signup going unidentified; fetchUserData reports the failure once.
  • Signup read deadline: the signup gives its user read 10 seconds in all, retries included, so a stalled /api/users/@me/ can't hold the signup up. fetchUserData (src/shared/api.ts) takes an optional signal that cancels the request in flight and cuts the backoff between retries short (retryTransient waits with node:timers/promises, taking the signal), and a cancelled request isn't retried. Its other callers pass no signal and wait as before. A read that fails on an untrusted certificate chain is diagnosed as on main, with a chain check of up to 5 seconds that the deadline doesn't cut short.

This revision (2026-11-13, takeover, 6e72625f to c6eb9143): merges 27 commits of main with no conflicts. A QA swarm on the merged head found that the tests asserted on mocked analytics and axios calls; they now check what PostHog ingestion and a local PostHog API receive. It also found that a failed signup user read was reported twice, which is now once, and replaced the hand-made abortable backoff timer with node:timers/promises. CodeQL flagged the test proxy's host lookup; it now answers each host from a switch. The proof judge then found that the 10-second deadline, which sat in fetchUserData, also failed OAuth and API-key logins whose user read answers after 10 seconds; a shadow diff against main confirmed it, so the deadline now covers only the signup's read.

Test plan

Lane gate on c6eb9143 (fleet heavy): pnpm typecheck green, pnpm lint 0 errors, pnpm test 4,268 tests in 280 files green. Full gate: CI on this PR (build, lint, unit tests on Node 22, 24 and 26, CodeQL, Semgrep, Wiz).

Test seams, each mutation-checked:

  • email-providers.test.ts: reportCodebaseScan(store) after a real scan of a temp project against the fixture catalog, with the real analytics client and a fake PostHog ingestion endpoint. A Resend + Stripe project, reported before and twice after login, delivers one wizard: email providers detected with ["Resend"], and no $set or $set_once on any ingested event names a provider. A Stripe-only project, a login without a user, and a declined-sharing session deliver none. Removing the user gate, the empty check, the latch or the consent check, or adding a $set, fails it.
  • signup.test.ts: getOrAskForProjectData({ signup: true }) with the real provisioning client and fetchUserData against a local PostHog API, and the real analytics client. A readable user is returned as the login's user and $identify reaches ingestion on its id. Each run loads a fresh analytics client, so it starts anonymous. A 403 still signs the account in, with no user, no $identify, one $exception, and no event on the new account's id; so does a user read that never answers, after 10 seconds. Removing the identify, the old double report, or the signup's deadline fails it.
  • api-errors.test.ts: the real fetchUserData and axios with a 3-second deadline signal, against a local server that never answers (rejects at the deadline), and, through a local HTTP proxy so the host counts as remote and retries, a host that answers 503 and then never answers (rejects within one deadline after two requests) and one whose second 503 lands just before the deadline (throws that 503 when the backoff crosses the deadline). Dropping the request's signal or the post-backoff deadline check fails them.

Proofs per qa-swarm PROOF.md, evidence in private proof storage:

  • Shadow diff on c6eb9143 against main (00c63a8b): the same harness on both sides over the same inputs, a --signup run whose user read answers, answers 403, answers after 12 seconds, or never answers, and the shared user read alone answering at once, after 12 seconds, or never. Every difference is named by the goal: on signup runs, $identify and $create_alias on the new account and one provider event when the read answers, and one $exception and an unidentified signup (after 10 seconds for the slow and silent reads) when it doesn't. The shared user read matches main on all three inputs, and the shared retry path without a deadline (user and project reads of a remote host that answers 503 then 200, or 503 every time) matches it in outcome, requests and timing.
  • UI walk on c6eb9143 against main: after the real signup, the real integration deck in the real LearnCard, rendered with ink-testing-library; the first frame reads Welcome. on main and Welcome, New. on this PR.
  • Live run on c6eb9143: the TUI's real oauthLogin and logIn on a --signup session, then reportCodebaseScan before and after login on a project with next, resend and stripe, scanned against the live public tool catalog, over real HTTP to a local stand-in for the provisioning API and /api/users/@me/. Analytics requests were logged and answered locally. User read answers: $identify and $create_alias on the new account, wizard: auth complete, then exactly one wizard: email providers detected with ["Resend"]; the report before login sent nothing, and no other event carried the providers. User read 403: the signup finished unidentified, one $exception, no provider event. User read after 12 seconds, or never: the signup finished unidentified after 10 seconds, one $exception.
  • Prod shape, flags: the Wizard's getAllFlagsForWizard against live PostHog on 2026-11-13, as an anonymous run and then after identifyUser for a new non-staff account, returned the same value for all eight flags; only /flags requests went out. Their release conditions explain it: person conditions name only PostHog staff, and every flag rolls out to everyone else at 100%.
  • Prod shape on bd339953, still valid on c6eb9143 (the scan and its event are unchanged since): the last 30 days of production wizard: warehouse sources detected events in the internal project report email kinds under exactly the names this PR lists (Resend, SendGrid, AwsSes, Brevo, Mailgun, Postmark, Mailchimp, Loops, MailerLite, Beehiiv, Mailjet seen as-is), and a clear minority of scans carry one. The counts stay in private proof storage. That event comes only from the interactive integration flow, so CI, warehouse and self-driving runs aren't in the sample.
  • Gap: no real account, OAuth login or ingestion. A real signup would create an account, and the client's analytics key and host are fixed.
  • Post-deploy watch: owed after release.
Captured events, signup live run on c6eb914
$identify                         signup-new-user  {"$set":{"email":"new@example.com","name":"New"},"$set_once":{}}
$create_alias                     signup-new-user
wizard: auth complete             signup-new-user  {"project_id":4242}
wizard: email providers detected  signup-new-user  {"email_provider_kinds":["Resend"],"project_id":4242}

The $identify is the existing login identify, with the account's own email and name. project_id is the run's existing tag. No captured request carries the providers anywhere else.

LLM context

Built by Claude Code (Opus 5.5) from a handoff in Michael's Workflows onboarding thread, and taken over on 2026-11-13 from the PR digest to bring it up to main. mayteio approved an earlier head.

Question Answer Who Where
Person property or event? An event only, never on the person, after the PII concern on #1493. Replaces the $set of two person properties 👤 Human comment
Should --signup runs identify the new account? Yes: read the new user with the provisioned token and identify it, as OAuth does. Replaces leaving signup runs anonymous as a follow-up. 👤 Human comment
Does identifying signup runs make this a one-way door that owes a shadow diff? Yes, once released: the person writes and alias merges stay. Graded high, covered by a shadow diff against main. Replaces the earlier "no". 🤖 Agent comment
Which user reads get the 10-second deadline? Only the signup's. OAuth and API-key logins and the AI opt-in screen wait as before; a deadline for them is a separate change. Replaces the deadline on every user read. 🤖 Agent comment
Email providers only, or a general wizard_detected_services? Email providers only. A general list costs the same to send but has no consumer yet, and the scan report already carries every detected kind. 🤖 Agent comment
Which kinds count as email providers? Services whose main job is sending email, marketing email included. Test inboxes (Mailtrap, Mailosaur) and multi-channel platforms (ActiveCampaign, Omnisend, Customer.io, Klaviyo, Braze, Iterable, Courier, Knock, OneSignal) stay out. 🤖 Agent comment
When does the event go out? When the scan is shared after login with a known user, for every program that scans, once per run. A run with no email provider sends nothing. 🤖 Agent comment

Moved from the fork PR #1493, where the review history and the decisions comment live.

Involvement: no grilling rounds, no prototypes, a design pass, a cross-family review, a QA swarm panel (7 lenses across GPT-6.1 Sol and Opus 5.5, including the author's Team perspective), a narrow second round on its fixes, and two final reads on #1493; on this PR a cross-family review (one finding, fixed), a clean final read, a GPT-6.1 Sol review on the event-only revision (one high and one low, both fixed), and a final read on its fixes (one high, fixed); live-run proof captured. On the takeover head: a QA swarm panel (8 lenses across GPT-6.1 Sol and Opus 5.5, including the author's Team perspective; 1 blocker and 4 smaller findings fixed, 1 rejected), a narrow second round (3 fixed), a proof judge (the user-read deadline scoped to signup, the provider test widened to every person write, the grade raised to high), a final read on that fix, PostHog Review on every head, and shadow-diff, live-run and prod-shape proof captured.

🤖 Generated with Claude Code

Silthus and others added 5 commits October 31, 2026 09:54
The codebase scan already finds email-sending services such as Resend.
Set them on the person as wizard_detected_email_providers and
wizard_detected_email_provider_at, through $set on the
`warehouse sources detected` event. A $set captured before login is set
again at identify, so it reaches the person who uses the PostHog app.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set the person properties when the scan is shared after login, so CI,
headless, warehouse and self-driving runs set them too, and drop the
pre-login replay in Analytics. Count Campaign Monitor, ConvertKit,
beehiiv and Drip as email providers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read the new account's user after provisioning and identify it, as the
OAuth login does, so a signup run's events and the detected email
providers land on its person. A failed read keeps the signup.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A login without a user, such as a signup whose user could not be read,
has no person to set them on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci feature-flags
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/google-adk
  • /wizard-ci ai-observability/groq
Show more apps
  • /wizard-ci ai-observability/manual-capture
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/opentelemetry
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci feature-flags/django
  • /wizard-ci feature-flags/next-js
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a Context Mill branch:

  • /wizard-ci all context-mill:my-branch

Add context-mill:<branch> to any command above to pin the Context Mill branch. It defaults to main.

Results will be posted here when complete.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus added this pull request to stack #1509 November 2, 2026 07:04
@Silthus
Silthus marked this pull request as ready for review November 2, 2026 07:05
@Silthus
Silthus requested a review from a team as a code owner November 2, 2026 07:05
…rson

Setting them on the person can track more than the user agreed to.
Send the kinds once per run on `wizard: email providers detected`
instead, after login and under the same sharing gates, and drop
setPersonProperties.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus

Silthus commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Note

🤖 Claude Opus 5.5 responding on behalf of Michael

Decisions for the event-only revision, b60ff263:

  • 👤 Person property or event? An event only, wizard: email providers detected with email_provider_kinds, never on the person, after the PII concern on feat(analytics): set detected email providers on the person #1493. Replaces the $set of wizard_detected_email_providers and wizard_detected_email_provider_at.
  • 🤖 When does the event go out? When the scan is shared after login with a known user, for every program that scans, once per run. A run with no email provider sends nothing. The event carries no scan time, since it has its own timestamp.

Silthus and others added 2 commits November 3, 2026 10:42
Each attempt had its own 10-second timeout, so a stalled remote read
ran about 32 seconds across its retries. Share one AbortSignal deadline
across the attempts, and don't retry a cancelled request. Drop a
duplicate assertion from the email provider event test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 10-second deadline cancelled each request but not the wait between
retries, so a late 503 plus 1.5 s of backoff ran past it. retryTransient
now takes the signal, cuts its backoff short when it aborts, and throws
the last failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@veria-ai

veria-ai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Reviews

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed d7af7a6: no findings.

…boundaries

The tests now check what PostHog ingestion and a local PostHog API receive,
not which mocked method ran: the providers event and its absence, the
signup's $identify, and the user read's deadline through a real stalled
server and a proxied remote host that retries.

A failed signup user read was reported twice, once by fetchUserData and
again by the signup; it is now reported once. The retry backoff uses
node:timers/promises instead of a hand-made abortable timer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed bd33995: no findings.

…ne test's margin

The ingestion tests await the analytics client's own flush instead of
polling for a sentinel event. The backoff deadline test now lands its
503 about 9 seconds in, before the 1.5-second backoff, so it holds half
a second of margin either side. The test proxy keys its behaviours by
a Map, so a request's host header cannot reach an object's prototype.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus

Silthus commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Note

🤖 Claude Opus 5.5 responding on behalf of Michael

Decisions from the takeover revision, d7af7a62 to c6eb9143:

  • 🤖 Does identifying signup runs make this a one-way door that owes a shadow diff? Yes, once the next Wizard release sends them: the $identify person writes and the $create_alias merge of the run's own pre-login id stay after the code is gone. Graded high; a shadow diff against main covers it, and every difference it shows is the intended identify. Replaces the earlier "no".
  • 🤖 Which user reads get the 10-second deadline? Only the signup's, which this PR adds. OAuth and API-key logins and the AI opt-in screen pass no deadline and wait as on main, so a read that answers after 10 seconds no longer fails their login. A deadline for those reads is a separate change.

CodeQL flagged the Map lookup too: it still picks a function by the
request's Host header. A switch over the two literal hosts picks no
function from input.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed a60d5fa: no findings.

The 10-second deadline lived in fetchUserData, so it also cut short the
user read of every OAuth and API-key login and the AI opt-in screen: a
read that answered after 10 seconds failed a login that used to finish.
Only the signup needs it, since its user read is new and must not hold
the signup up. fetchUserData now takes an optional signal; the signup
passes its 10-second deadline, and every other caller waits as before.

The provider test now checks every person write ingestion receives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed c6eb914: no findings.

@Silthus Silthus added stamphog Auto-merge stamp for automated PRs and removed stamphog Auto-merge stamp for automated PRs labels Nov 13, 2026
The failure cases reused the analytics client the successful signup had
identified, so they could not show their events stay anonymous. Load the
client fresh per run and assert no event lands on the new account.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus Silthus changed the title feat(analytics): set detected email providers on the person feat(analytics): report detected email providers and identify signup runs Nov 13, 2026

@posthog posthog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PostHog Review · Standard · reviewed 3e52769: no findings.

@Silthus
Silthus requested a review from mayteio November 13, 2026 06:36
@Silthus
Silthus merged commit b311f2f into main Nov 13, 2026
22 checks passed
@Silthus
Silthus deleted the feat/detected-email-providers branch November 13, 2026 14:15
@mon2000x111-sketch

Copy link
Copy Markdown

Silthus added a commit that referenced this pull request Nov 18, 2026
…ffer

Takes main's user-read deadline from the squash of #1506 in api.ts,
api-errors.test.ts and project-data.ts, and keeps both new flag keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silthus added a commit that referenced this pull request Nov 18, 2026
main tests the detected-providers event in email-providers.test.ts and
the signup identify in signup.test.ts, so these copies only duplicate
them in this PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants