Repository navigation
feat(analytics): report detected email providers and identify signup runs - #1506
Merged
Merged
Conversation
The codebase scan already finds email-sending services such as Resend. Set them on the person as wizard_detected_email_providers and wizard_detected_email_provider_at, through $set on the `warehouse sources detected` event. A $set captured before login is set again at identify, so it reaches the person who uses the PostHog app. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Set the person properties when the scan is shared after login, so CI, headless, warehouse and self-driving runs set them too, and drop the pre-login replay in Analytics. Count Campaign Monitor, ConvertKit, beehiiv and Drip as email providers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Read the new account's user after provisioning and identify it, as the OAuth login does, so a signup run's events and the detected email providers land on its person. A failed read keeps the signup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A login without a user, such as a signup whose user could not be read, has no person to set them on. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Test against a Context Mill branch:
Add Results will be posted here when complete. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rson Setting them on the person can track more than the user agreed to. Send the kinds once per run on `wizard: email providers detected` instead, after login and under the same sharing gates, and drop setPersonProperties. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Note 🤖 Claude Opus 5.5 responding on behalf of Michael Decisions for the event-only revision,
|
Each attempt had its own 10-second timeout, so a stalled remote read ran about 32 seconds across its retries. Share one AbortSignal deadline across the attempts, and don't retry a cancelled request. Drop a duplicate assertion from the email provider event test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 10-second deadline cancelled each request but not the wait between retries, so a late 503 plus 1.5 s of backoff ran past it. retryTransient now takes the signal, cuts its backoff short when it aborts, and throws the last failure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mayteio
approved these changes
Nov 4, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…boundaries The tests now check what PostHog ingestion and a local PostHog API receive, not which mocked method ran: the providers event and its absence, the signup's $identify, and the user read's deadline through a real stalled server and a proxied remote host that retries. A failed signup user read was reported twice, once by fetchUserData and again by the signup; it is now reported once. The retry backoff uses node:timers/promises instead of a hand-made abortable timer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ne test's margin The ingestion tests await the analytics client's own flush instead of polling for a sentinel event. The backoff deadline test now lands its 503 about 9 seconds in, before the 1.5-second backoff, so it holds half a second of margin either side. The test proxy keys its behaviours by a Map, so a request's host header cannot reach an object's prototype. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Note 🤖 Claude Opus 5.5 responding on behalf of Michael Decisions from the takeover revision,
|
CodeQL flagged the Map lookup too: it still picks a function by the request's Host header. A switch over the two literal hosts picks no function from input. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 10-second deadline lived in fetchUserData, so it also cut short the user read of every OAuth and API-key login and the AI opt-in screen: a read that answered after 10 seconds failed a login that used to finish. Only the signup needs it, since its user read is new and must not hold the signup up. fetchUserData now takes an optional signal; the signup passes its 10-second deadline, and every other caller waits as before. The provider test now checks every person write ingestion receives. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The failure cases reused the analytics client the successful signup had identified, so they could not show their events stay anonymous. Load the client fresh per run and assert no event lands on the new account. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silthus
added a commit
that referenced
this pull request
Nov 18, 2026
…ffer Takes main's user-read deadline from the squash of #1506 in api.ts, api-errors.test.ts and project-data.ts, and keeps both new flag keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Silthus
added a commit
that referenced
this pull request
Nov 18, 2026
main tests the detected-providers event in email-providers.test.ts and the signup identify in signup.test.ts, so these copies only duplicate them in this PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The Wizard's codebase scan already finds the email services a user's code sends through, such as Resend, SendGrid or Amazon SES. That fact stays in the scan, so we can't count how many Wizard users already send email from code, or follow them into Workflows.
With this PR, a run that finds an email provider sends one
wizard: email providers detectedevent withemail_provider_kinds(the detected kinds, such as["Resend"]). Nothing is set on the person. This covers every program that scans: the default integration flow, interactive or CI,wizard warehouseandwizard self-driving. Awizard --signuprun now identifies the account it creates, so its events land on that account's person too.Impact
Pays into Grow Workflows through Started orgs (orgs that create their first workflow). Not measured yet: an insight in the internal project counts people who sent
wizard: email providers detectedper week, and how many of them reach Workflows, start its onboarding, create an email sender, or launch a workflow or broadcast. In scope: email, the Messaging category. Expected move: small, up. On its own this PR only adds the signal. A feature flag can't target an event, so a follow-up that targets these users, such as a "We found Resend in your code" card in Workflows, needs a cohort or its own check. Run cost: n/a.Merge Danger
Caution
🔴 Merge risk: high · one-way door once the next Wizard release: the person writes and alias merges of signup runs stay
wizard --signupruns, in both regions, now read the new account's user and identify it:$identifysets that account's own email and name on its person, and$create_aliasmerges the run's pre-login id into it, as OAuth and API-key logins already do throughlogIn. Those writes stay once sent. A failed or slow read leaves the signup going unidentified after at most 10 seconds, with one exception. Covered by the shadow diff (read answers, 403, after 12 seconds, never) andsignup.test.ts. Uncovered row: a read that fails on an untrusted certificate chain just before the deadline still runs the chain checkmainruns on every user read, with its own 5-second limit, so that signup goes on up to 5 seconds later.Welcome, New.where it saidWelcome.), their later events carry the new account's organization group, and the Wizard's feature flags are evaluated as that user. Every flag resolves as before for a new account: the Wizard's own flag evaluation against live PostHog returned the same value for all eight flags, anonymous and identified. The AI opt-in gate still skips signup runs. Covered by the shadow diff, including the real learn card's first frame.wizard: email providers detectedevent with the provider kinds. Nothing is written to the person. Covered by the shadow diff, the live run (a signup run only) andemail-providers.test.ts, which checks every person write but seeds the logged-in user instead of identifying the analytics client. The post-deploy watch then confirmed in production that the event lands on identified runs after the other logins too.mainfor a user read that answers at once, after 12 seconds, or never, and for user and project reads of a remote host that recover on a retry or run out of retries, with the same requests and timing.Rollout
$exceptionfromfetchUserDataand theirwizard: auth completein the Wizard's analytics project, for a week after the release. No alert covers either.Changes
detectedEmailProviders(src/services/codebase-scan/email-providers.ts) reads the run's codebase scan and keeps the tool catalog kinds of email services:AwsSes,Beehiiv,BigMailer,Brevo,CampaignMonitor,ConvertKit,Drip,Elasticemail,EmailOctopus,Flexmail,Loops,Mailchimp,MailerLite,MailerSend,Mailgun,Mailjet,Plunk,Postmark,Resend,SendGrid,Shipmail,Smaily,SparkPost,Tinyemail. Kind names match PostHog's live/api/wizard/tool_catalog/. The kinds are a typed list,EmailProviderKind.reportCodebaseScan, the one place a scan is cleared to leave the machine, captureswizard: email providers detectedwhen it shares the scan: sharing granted, telemetry on, logged in with a known user, and the scan did not fail. A login without a user (a signup whose user read failed, or an API key withoutuser:read) sends nothing. The scan-share latch keeps it to once per run. A scan with no email provider sends nothing. The event carries only the provider kind names, from the fixed list above.askForProvisioningSignup(src/tui/auth/project-data.ts) reads the new account's user with the provisioned token, which already holdsuser:read, identifies it, and returns it as the login's user. Before, a signup run never identified anyone, so every event stayed on an anonymous person. A failed read leaves the signup going unidentified;fetchUserDatareports the failure once./api/users/@me/can't hold the signup up.fetchUserData(src/shared/api.ts) takes an optional signal that cancels the request in flight and cuts the backoff between retries short (retryTransientwaits withnode:timers/promises, taking the signal), and a cancelled request isn't retried. Its other callers pass no signal and wait as before. A read that fails on an untrusted certificate chain is diagnosed as onmain, with a chain check of up to 5 seconds that the deadline doesn't cut short.This revision (2026-11-13, takeover,
6e72625ftoc6eb9143): merges 27 commits ofmainwith no conflicts. A QA swarm on the merged head found that the tests asserted on mockedanalyticsandaxioscalls; they now check what PostHog ingestion and a local PostHog API receive. It also found that a failed signup user read was reported twice, which is now once, and replaced the hand-made abortable backoff timer withnode:timers/promises. CodeQL flagged the test proxy's host lookup; it now answers each host from aswitch. The proof judge then found that the 10-second deadline, which sat infetchUserData, also failed OAuth and API-key logins whose user read answers after 10 seconds; a shadow diff againstmainconfirmed it, so the deadline now covers only the signup's read.Test plan
Lane gate on
c6eb9143(fleet heavy):pnpm typecheckgreen,pnpm lint0 errors,pnpm test4,268 tests in 280 files green. Full gate: CI on this PR (build, lint, unit tests on Node 22, 24 and 26, CodeQL, Semgrep, Wiz).Test seams, each mutation-checked:
email-providers.test.ts:reportCodebaseScan(store)after a real scan of a temp project against the fixture catalog, with the real analytics client and a fake PostHog ingestion endpoint. A Resend + Stripe project, reported before and twice after login, delivers onewizard: email providers detectedwith["Resend"], and no$setor$set_onceon any ingested event names a provider. A Stripe-only project, a login without a user, and a declined-sharing session deliver none. Removing the user gate, the empty check, the latch or the consent check, or adding a$set, fails it.signup.test.ts:getOrAskForProjectData({ signup: true })with the real provisioning client andfetchUserDataagainst a local PostHog API, and the real analytics client. A readable user is returned as the login's user and$identifyreaches ingestion on its id. Each run loads a fresh analytics client, so it starts anonymous. A 403 still signs the account in, with no user, no$identify, one$exception, and no event on the new account's id; so does a user read that never answers, after 10 seconds. Removing the identify, the old double report, or the signup's deadline fails it.api-errors.test.ts: the realfetchUserDataand axios with a 3-second deadline signal, against a local server that never answers (rejects at the deadline), and, through a local HTTP proxy so the host counts as remote and retries, a host that answers 503 and then never answers (rejects within one deadline after two requests) and one whose second 503 lands just before the deadline (throws that 503 when the backoff crosses the deadline). Dropping the request's signal or the post-backoff deadline check fails them.Proofs per qa-swarm
PROOF.md, evidence in private proof storage:c6eb9143againstmain(00c63a8b): the same harness on both sides over the same inputs, a--signuprun whose user read answers, answers 403, answers after 12 seconds, or never answers, and the shared user read alone answering at once, after 12 seconds, or never. Every difference is named by the goal: on signup runs,$identifyand$create_aliason the new account and one provider event when the read answers, and one$exceptionand an unidentified signup (after 10 seconds for the slow and silent reads) when it doesn't. The shared user read matchesmainon all three inputs, and the shared retry path without a deadline (user and project reads of a remote host that answers 503 then 200, or 503 every time) matches it in outcome, requests and timing.c6eb9143againstmain: after the real signup, the real integration deck in the realLearnCard, rendered with ink-testing-library; the first frame readsWelcome.onmainandWelcome, New.on this PR.c6eb9143: the TUI's realoauthLoginandlogInon a--signupsession, thenreportCodebaseScanbefore and after login on a project withnext,resendandstripe, scanned against the live public tool catalog, over real HTTP to a local stand-in for the provisioning API and/api/users/@me/. Analytics requests were logged and answered locally. User read answers:$identifyand$create_aliason the new account,wizard: auth complete, then exactly onewizard: email providers detectedwith["Resend"]; the report before login sent nothing, and no other event carried the providers. User read 403: the signup finished unidentified, one$exception, no provider event. User read after 12 seconds, or never: the signup finished unidentified after 10 seconds, one$exception.getAllFlagsForWizardagainst live PostHog on 2026-11-13, as an anonymous run and then afteridentifyUserfor a new non-staff account, returned the same value for all eight flags; only/flagsrequests went out. Their release conditions explain it: person conditions name only PostHog staff, and every flag rolls out to everyone else at 100%.bd339953, still valid onc6eb9143(the scan and its event are unchanged since): the last 30 days of productionwizard: warehouse sources detectedevents in the internal project report email kinds under exactly the names this PR lists (Resend,SendGrid,AwsSes,Brevo,Mailgun,Postmark,Mailchimp,Loops,MailerLite,Beehiiv,Mailjetseen as-is), and a clear minority of scans carry one. The counts stay in private proof storage. That event comes only from the interactive integration flow, so CI, warehouse and self-driving runs aren't in the sample.Captured events, signup live run on c6eb914
The
$identifyis the existing login identify, with the account's own email and name.project_idis the run's existing tag. No captured request carries the providers anywhere else.LLM context
Built by Claude Code (Opus 5.5) from a handoff in Michael's Workflows onboarding thread, and taken over on 2026-11-13 from the PR digest to bring it up to
main. mayteio approved an earlier head.$setof two person properties--signupruns identify the new account?main. Replaces the earlier "no".wizard_detected_services?Moved from the fork PR #1493, where the review history and the decisions comment live.
Involvement: no grilling rounds, no prototypes, a design pass, a cross-family review, a QA swarm panel (7 lenses across GPT-6.1 Sol and Opus 5.5, including the author's Team perspective), a narrow second round on its fixes, and two final reads on #1493; on this PR a cross-family review (one finding, fixed), a clean final read, a GPT-6.1 Sol review on the event-only revision (one high and one low, both fixed), and a final read on its fixes (one high, fixed); live-run proof captured. On the takeover head: a QA swarm panel (8 lenses across GPT-6.1 Sol and Opus 5.5, including the author's Team perspective; 1 blocker and 4 smaller findings fixed, 1 rejected), a narrow second round (3 fixed), a proof judge (the user-read deadline scoped to signup, the provider test widened to every person write, the grade raised to high), a final read on that fix, PostHog Review on every head, and shadow-diff, live-run and prod-shape proof captured.
🤖 Generated with Claude Code