Skip to content

feat(analytics): ask about the email provider after install - #2

Closed
Silthus wants to merge 10 commits into
feat/detected-email-providersfrom
feat/email-provider-offer
Closed

Silthus wants to merge 10 commits into
feat/detected-email-providersfrom
feat/email-provider-offer

Conversation

@Silthus

@Silthus Silthus commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Stacked on PostHog#1493 (base feat/detected-email-providers), which is still open, so the base stays. Restacked on main on 2026-10-31 with no conflicts: the base branch moved onto main, then this PR onto the base. Fork-only slice 18a of the Wizard email work; it mirrors no upstream PR yet.

This revision: QA swarm fixes on 2026-11-01. d6d07fdc asks only once the run identified the user, test-first. d5b7286b bounds the Ctrl-C and signal report by the same 2-second budget as a screen's exit request, test-first. 8b06cbd9 keeps each offered provider's kind and label together in one list. The live run and UI walk below ran on 324302a. These commits change two things they did not see: a run that could not identify the user gets no question, and every Ctrl-C or signal, mid-run included, now waits at most 2 seconds for its analytics report, so on a slow network the cancelled setup wizard finished event can be lost where the process used to wait for the analytics client's own timeout. The new tests cover both.

Problem

The Wizard's scan finds the email service a developer's app sends with, such as Resend. Today nothing asks them about it, so we can't tell how many of them would send email from PostHog instead, or with what intent.

With this PR, an interactive install that found an email provider can ask one question after the outro: "Your app sends email with Resend. PostHog can send it too, triggered by the events you already track. Want to try it?" The answer goes to analytics and onto the person. The question only shows while the flag wizard-email-provider-offer is on for the run, and the flag starts at 0%.

Spec slice 18 reuses this prompt for its migrate resend offer: the "Yes" screen already has a typed slot for the migrate command, which stays empty until slice 16's migrate family exists.

Impact

Pays into Grow Workflows through Started orgs (orgs that create their first workflow). Measured by the funnel Wizard email provider offer in the internal project: offer shown, answered (by answer), then a Workflows page view or an email sender created within 7 days, broken down by providers and answer. It counts production builds only, and only runs where wizard-email-provider-offer is on. Expected move: small, up. The first job of this PR is the intent read; the "Yes" link is the conversion path. Run cost: n/a.

Merge Danger

Note

🔵 Merge risk: low · one-way door (merge-risk rubric)

  • Released Wizards write wizard_email_offer_answer and wizard_email_offer_answered_at onto real person profiles once the flag is on. A revert stops new writes; values already set stay until a $unset.
  • A new flow step sits after the outro of every interactive run. It shows only when the flag is on, the run succeeded, sharing is on, and the scan found an email provider. The flag starts at 0%, so merging changes nothing a user sees.
  • Ctrl-C and process signals now unmount the TUI first, then wait up to 2 seconds for the analytics flush before they exit. That path is shared by every interactive run.

Changes

  • Flag (WIZARD_EMAIL_PROVIDER_OFFER_FLAG_KEY in src/shared/constants.ts): wizard-email-provider-offer joins the Wizard's closed flag set, WIZARD_FLAG_KEYS. The run already evaluates that set once after login and caches it. The offer reads the cached snapshot, so it adds no flag request.
    • Only the value 'true' asks. A flag that is off, missing from the snapshot, or never evaluated means no question.
    • A failed flag request already leaves an empty snapshot, so it means no question too. The run never waits on the flag or fails because of it.
  • When it asks (emailProviderOffer in src/programs/posthog-integration/email-provider-offer.ts): the flag is on, the run is interactive and not --ci, the run identified the user, scan results may be shared (sharing granted and telemetry on), the outro reports success, and the scan found at least one email provider. Otherwise it returns nothing and the step never shows. Headless runs never mount the TUI, so they never reach it.
  • The question (EmailProviderOfferScreen, a new email-provider-offer step between the outro and the MCP step):
    • It names up to two providers: "with Resend", "with Resend and SendGrid", or "with Resend, SendGrid and others".
    • "Yes, show me how": shows "Set up email in Workflows: <app host>/project/<id>/workflows?source=wizard_cli" and moves on at the next key. The link stays on screen only.
    • "Not now" moves on at once. So does choice 3: "No, I'm happy with Resend" for one provider, or "No, I'm happy with my current setup" for several.
    • Esc, or Ctrl-C while the question is up, counts as skipped.
    • It asks once per run; a second answer or an unmount after an answer sends nothing.
  • Events, through the existing wizardCapture helper:
    • wizard: email provider offer shown with providers: every detected kind, such as ["Resend", "SendGrid"], even when the prompt names only two.
    • wizard: email provider offer answered with providers, answer (yes, not_now, no, skipped), and $set of wizard_email_offer_answer and wizard_email_offer_answered_at. The run is logged in by then, so the $set lands on the identified person.
  • Ctrl-C and signals deliver the answer (src/tui/run.ts): after a successful run the run report has already gone out, so the interrupt's analytics.shutdown('cancelled') returned at once and the process could exit before the skipped event left. The signal handler now unmounts the TUI first, which captures skipped for an open question, then awaits the shutdown and a flush, for at most the 2-second report budget, before it exits. The screen-exit path already used that order and budget.
  • Provider list: the offer holds one non-empty list of { kind, label } pairs; the copy reads the labels and the events send the kinds. detectedEmailProviders(scan) in email-providers.ts now backs both feat(analytics): set detected email providers on the person PostHog/wizard#1493's person property and this offer, so the two can't drift. The kind list became a typed tuple, so EmailProviderKind keys the migrate command map.

Test plan

Lane gate on 8b06cbd9: pnpm typecheck green, pnpm lint 0 errors, pnpm exec vitest run 261 files and 3,888 tests green. Earlier, on 324302a: 258 files and 3,838 tests. CI: the fork runs no workflows, so this PR has no checks; Michael accepted that gap, and the lane gate is the check.

Test seams:

  • emailProviderOffer(session, flags) over a real scan session (email-provider-offer.test.ts):
    • The flag on gives the offer with the Workflows link and source=wizard_cli.
    • The flag off, missing from the snapshot, or with no snapshot gives no offer.
    • No offer for --ci, telemetry off, sharing declined, a failed install, no login, a login that could not identify the user, and a scan with no email provider. The identity case was red before d6d07fdc.
    • The copy for one, two and three providers, with providers keeping all of them.
  • Analytics.getAllFlagsForWizard (analytics.test.ts): a rejected flag request caches a snapshot with no flags, which the offer reads as off.
  • EmailProviderOfferScreen rendered in the real store with @utils/analytics mocked (EmailProviderOfferScreen.test.tsx): the exact prompt copy and the shown event; not_now, no and Esc; Yes shows the link and the next key moves on; one answer only; an unmount without an answer sends skipped.
  • runTui cancellation (run.test.ts):
    • An Ink Ctrl-C resolves 130 only after the analytics flush finishes.
    • A SIGINT, SIGTERM or SIGHUP unmounts the TUI before that final flush.
    • After a successful run, an Ink Ctrl-C flushes the answer the unmount captures before it resolves 130. This case fails on the earlier flush-then-unmount order.
    • After a successful run, an Ink Ctrl-C resolves 130 within the 2-second budget when the flush hangs. Red before d5b7286b (the test timed out).
  • Flow, frame, store and harness registries pick up the new step and flag key (flows.test.ts, frames.test.tsx snapshots at 80x28 and 120x40, store-invariants.test.ts, flags.test.ts, the e2e action registry). The analytics mocks in four flow tests and the harness test gained the flag read.

Proofs per qa-swarm PROOF.md, captured on 324302a on another machine; this box has no copy, so on this head they are gaps that point at these captures. 324302a adds only a test, so the captures from a71b308 exercise the same source.

  • Live run (saved): the real TUI in a PTY on a Next.js fixture, after the real scan and detection, a real login through Analytics, and the real flag evaluation against the internal project.

    • Flag off: the flag evaluated to false at 0%. The outro went straight to exit, with no offer screen and no offer events.

      Flag off: after the outro the run exits

    • Flag on, two providers: the Wizard's dev-only flag override turned the flag on, on a fixture with resend and @sendgrid/mail. The prompt read "Your app sends email with Resend and SendGrid." and choice 3 read "No, I'm happy with my current setup". "Yes" showed the Workflows link, and shown and answered carried ["Resend", "SendGrid"] with answer: yes.

      Flag on, two providers: the question

      After Yes: the Workflows link

    • Each answer, flag on, one provider. Each sent shown then answered to the internal project, and the person got wizard_email_offer_answer:

      keys after the outro screen answer sent
      Enter on "Yes, show me how" the Workflows link, then exit on the next key yes
      Down, Enter exit not_now
      Down, Down, Enter exit no
      Esc exit skipped
      Ctrl-C exit 130 skipped

      One provider, choice 3 selected

    • Ctrl-C after a successful run, on 324302a: the harness sent the real success shutdown before the question, as a successful run does. Ctrl-C then ran the host's signal order: unmount, the cancelled shutdown (a no-op by then), the flush, and exit 130 with nothing after it. The internal project received setup wizard finished (success), then shown, then answered with skipped, and the person got wizard_email_offer_answer. So the flush alone delivers an answer captured after the run already reported its end.

      After a successful run: the question before Ctrl-C

  • CI case (saved): flag on and session.ci set. The outro goes straight to exit, with no offer screen and no offer events.

  • Prod shape (gap, accepted by standing rule until 2026-11-02): how many real runs find an email provider stays unknown until read-only production access is set up.

  • UI walk (saved): PNG captures of the real TUI at 100x30, shown above: flag off, two providers, the link, one provider, the CI run, and the question after a successful run.

  • Post-deploy watch: owed after merge.

  • Gap: no LLM agent install ran, since this box has no gateway key. The harness arranges the successful run and outro; the scan, detection, login, flag evaluation, TUI and analytics client are the real ones. For the same reason, the live Ctrl-C runs the host's steps in its order but not the runTui function itself: the repo's real-host route needs --ci, which hides the question. The run.test.ts cases drive runTui itself.

Decisions

  • 👤 Ask about the email provider in the Wizard, ahead of migrate? Yes: one question after the install outro, measured, never in CI or headless runs or with sharing off.
  • 👤 What does the question say? The prompt and three choices as Michael wrote them; Esc and Ctrl-C count as skipped.
  • 👤 What gates the question? A dedicated flag, wizard-email-provider-offer, at 0%. Not the Workflows launch flag: that one stays on the PostHog org until launch and would hide the question from the developers it measures.
  • 👤 What happens when the flag can't be read? No question, and the run carries on.
  • 👤 With several providers, what does the prompt name? Up to two: "Resend and SendGrid", or "Resend, SendGrid and others". Choice 3 then reads "No, I'm happy with my current setup". It replaced the agent's first-provider-only copy.
  • 👤 Does the empty migrate command slot stay? Yes, as the typed seam slice 16 fills; the review's call to remove it was rejected.
  • 👍 Where does the "Yes" link go? On screen only, before the MCP step, not in the exit summary.
  • 🤖 Who can be asked? Only a user the run identified, so the answer's $set never lands on an anonymous person (pending approval).
  • 🤖 Simplify Analytics.shutdown to always flush? Not in this PR: it changes every host's terminal path; a follow-up (pending approval).
  • 🤖 How does the Wizard read the flag? From the run's cached flag snapshot, by adding the key to the closed flag set, so no new request.
  • 🤖 What counts as reaching Workflows in the funnel? A page view under /project/<id>/workflows or an email integration created, within 7 days; production builds only.

Open: a quick second Enter after the outro answers yes, because the question opens with the cursor on "Yes, show me how", which can inflate the yes share. Recommended: ignore keys on the question for a moment after it mounts, before the flag rolls past 0%; it keeps the copy and the order. The alternative, starting the cursor on "Not now", changes the choices as written.

Follow-ups:

  • Analytics.shutdown could guard only the terminal event and always flush, which would let the TUI host drop its extra flush; it is shared by every host.
  • Tests that prove delivery through the real Analytics with a recording transport, in place of mocked flushes, for every host's terminal path.

LLM context

Built by Claude Code (Opus 5.5) as lane wiz-offer of the Wizard email map. Michael answered the open questions on 2026-10-26.

Involvement: no grilling rounds, no prototypes, one cross-family review (GPT-6.1 Sol, round 1: one high finding fixed, one low finding rejected), final reads of each later change, and two fixes from the proof judge (unmount before the final flush on a signal, and a test and live run for Ctrl-C after a successful run). Live-run and UI-walk proof captured. QA swarm 2026-11-01: one two-family panel round (10 lenses, GPT-6.1 Sol and Claude Opus 5.5) and a fixes read; 4 findings fixed, 2 rejected, 2 follow-ups, 1 open decision.

🤖 Generated with Claude Code

Silthus and others added 7 commits October 31, 2026 09:54
An interactive install run whose scan found an email provider now asks
one question after the outro: try sending email with PostHog Workflows?
The answer goes out as `wizard: email provider offer answered` and is
set on the person. Yes shows a link to Workflows in the project.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus
Silthus force-pushed the feat/detected-email-providers branch from 7e96c5d to ae686b9 Compare October 31, 2026 10:13
@Silthus
Silthus force-pushed the feat/email-provider-offer branch from 324302a to 90cccfc Compare October 31, 2026 10:14
Silthus and others added 3 commits November 1, 2026 18:45
…entified

A signup whose user read failed logs in without identifying the user, so
the answer and its person properties would land on the anonymous person.
The offer now also needs the session's user.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Ctrl-C or signal after a successful run now waits for its analytics
flush, so a slow network could hold the process after the terminal is
restored. The signal path races the report against the same budget as a
screen's exit request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…l together

The offer held provider kinds and labels in two arrays that had to stay in
step. It now holds one non-empty list of kind and label pairs; the copy
reads the labels and the events send the kinds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Silthus

Silthus commented Nov 2, 2026

Copy link
Copy Markdown
Owner Author

Moved to PostHog#1508.

@Silthus Silthus closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant