Skip to content

Fix what the eval and the client trial found - #22

Merged
franco-zalamena-iterable merged 12 commits into
mainfrom
skill/eval-and-trial-fixes
Sep 25, 2026
Merged

franco-zalamena-iterable merged 12 commits into
mainfrom
skill/eval-and-trial-fixes

Conversation

@franco-zalamena-iterable

@franco-zalamena-iterable franco-zalamena-iterable commented Sep 21, 2026 •

Copy link
Copy Markdown
Contributor

Twelve commits from two rounds of testing the Android and React Native skills against real apps: a 12-arm eval on a greenfield app, a non-greenfield scenario where the app already sent its own push, and a client trial in which an agent playing a protective app developer integrated the SDK end to end until a real push landed on a device. Rebased onto main — the two Xcode 27 commits this branch used to carry reached main via #21. Version ends at 26.9.3-beta.

Security and correctness

Refuse on-device JWT signing (ebbfecf). Told "JWT is on, here is the shared secret, we have no backend", the skill implemented HMAC signing in the app. Rule 1 said "wire up the handler" without saying where tokens come from, its pointer to the JWT doc was dangling, and the corpus ships a complete IterableJwtGenerator.java with nothing marking it as server code. Rule 1 now makes a backend a hard requirement and says to stop and stub without one; pitfall #22 names the corpus sample as the attractive nuisance it is.

Forward to an existing FirebaseMessagingService, don't replace it (0eec62b). Iterable's service registers at priority -1, so adding the SDK doesn't steal dispatch from an app that already sends order-status or security-code push — but Iterable receives nothing until their service forwards to it. The tempting fixes (delete their service, repoint the manifest, tools:node="remove") trade a silent marketing failure for a broken transactional channel. Android rule 8 + pitfall #23, RN rule 8 + pitfall #15, and Step 0 now greps for an existing service before scoping push work.

Accuracy of what the skill tells developers

Read dashboard nav paths, don't recite them (ebbfecf). The skill said Integrations > API Keys in one eval arm and Settings > API Keys in two others, while its own corpus gives the correct path in nine places. The routing table now carries the verified path per slug, with an instruction not to paraphrase and a rule that the doc wins on disagreement.

Add the dashboard prerequisite articles (f750f0a). api-keys, jwt-enabled-api-keys, fcm-http-v1-migration and the two embedded-messages articles, for both platforms — the steps a developer who doesn't yet have a key or a push integration has to perform.

Reconcile the init-callback guidance (c4620c1). PITFALLS.md #2 said the init callback should log initialization and nothing else; the canonical pattern said keep it empty and shipped a comment-only body. The trial followed the pattern — correctly, it's the one labelled "start here" — and read "should log" as optional. The pattern is now concrete: the SDK's verbose output proves the SDK initialized, but only the callback's own line separates "init never fired" from "init fired, identify didn't".

What the client trial cost

~43 minutes, and four of those causes are now closed.

  • Pitfall [SDK-425] Add React Native skill skeleton and multi-platform docs refresh #16 was Groovy-only (553ceef). Kotlin DSL is the default for new projects and the block doesn't compile there. It also needs buildFeatures { buildConfig = true }, which AGP defaulted on before 8.0; measured on AGP 8.4, the failure text is quoted verbatim because it reads as "the field is wrong" rather than "the flag is off".
  • Stop trusting the handed-over inputs (553ceef). The trial found a stale ITERABLE_API_KEY in local.properties belonging to another project, and caught it only by diffing against the key it had been given — nothing told it to look. Same shape for google-services.json against applicationId.
  • A definition of done that needs no dashboard (fd057d2, Android only). SKILL.md stopped at "initialized, identified, key wired, builds" while the push doc ends at Test Push, which needs access many app developers don't have. Three device-side checks replace the guesswork. One exists purely because the trial hit it: registerDeviceToken returned Success and getMessages immediately returned UnknownEmailError for the same address. The profile hadn't propagated, and repairing a non-defect is how a working integration gets broken.
  • Identity that arrives after init (42a104c). The largest single cost, 15 minutes. The canonical pattern took identity as a String, assuming it exists at Application.onCreate(); the app kept email in a DataStore Flow that emits at login and on account switch. The variant takes the stream and names the two shortcuts that look like simplifications — hoisting setEmail into the init callback (Package as a Cursor plugin (skill + Context7 MCP) #2), and first()-ing the flow back into a String, which drops the account switch (Claim Context7 library and fix MCP config #3).

Two independent reviews of this PR

Fourteen defects between them, all fixed (669d579, 5bc03bc, a196094, 4eb1d3a). The ones that mattered:

  • Both citations of the forwarding procedure gave its heading as ## Handling Firebase push messages and tokens. It is ####, nested under ### Step 5 — so an agent grepping at the level it was handed finds nothing, skips the procedure, and hits the failure rule 8 exists to prevent. Every doc anchor in both skills has since been checked against the real heading level.
  • The rebase renumbered this branch's new RN pitfall to Stop Context7 serving anti-pattern samples; collapse three doc copies into one #15 but left its label reading 14, which after the rebase is an Expo- and iOS-only item. The Android POST_NOTIFICATIONS note cited pitfall Reduce context7.json to minimal claim file #4 (postDelayed sequencing) instead of Enable live Context7 fetching (snapshot becomes fallback) #7.
  • The stale-key bullet offered a 401 as its confirmation signal. A key belonging to a different Iterable project authenticates normally and returns 200, quietly populating that project — so an agent watching for a 401 would read the stale key as verified. It now names the real symptom: none of it appears in their dashboard.
  • The definition of done claimed a notification channel observable before any push arrived. registerChannelIfEmpty has a single call site, inside the notification-building path, so it cannot be there yet; that step is dropped and the push-arrived check proves strictly more.

Verification

  • pnpm check:all green: typecheck, 37 reference docs against the schema, plugin manifests. 26.9.3-beta consistent across all three manifests.
  • Every pitfall #N cross-reference and doc anchor in both trees resolves to something that exists — checked by hand, because CI covers neither. A gate for it is a follow-up, as is the RN definition of done, which still has no device-side proof path.
  • The AGP 8.4 failure text, the dumpsys notification arrival output and the Iterable-side token registration were observed on a live emulator. The channel-creation claim was read from SDK source; the 401-versus-200 split is server behaviour, inferred from the SDK labelling any non-JWT 401 Invalid API Key.

🤖 Generated with Claude Code

The skill could explain SDK code but not the Iterable-side inputs that code
depends on, so agents recited dashboard navigation from memory and got it
wrong. Five articles cover those inputs: API key creation and types, JWT-enabled
keys, the FCM HTTP v1 migration, and the two embedded-messaging setup pages.

They document Iterable's web app rather than the SDK, which the existing three
archetypes had no room for, hence `prerequisite`. Being platform-neutral, both
android.yml and react-native.yml pull the same source paths.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…them

A 12-arm eval against a greenfield app surfaced two failures the skill caused
rather than prevented.

**On-device JWT signing.** Told "JWT is on, here is the shared secret, we have
no backend," the skill implemented HMAC signing in the app — documenting the
danger at length and complying anyway. Rule 1 said "wire up the handler" and
said nothing about where tokens come from, its pointer to the JWT doc was
dangling, and the bundled corpus ships a complete `IterableJwtGenerator.java`
with no note that it is server code. Every signpost pointed at the foot-gun.

Rule 1 now makes the backend a hard requirement and says to stop and stub when
one does not exist; pitfall #22 names the corpus sample as the attractive
nuisance it is. A build that cannot authenticate yet is recoverable in an
afternoon — a leaked signing secret means rotating it in Iterable and shipping
a new release.

**Dashboard navigation.** The skill said `Integrations > API Keys` in one
scenario and `Settings > API Keys` in two others, while its own corpus gives the
correct path in nine places. It recited from memory. The routing table now
carries the verified path per slug, with an instruction not to paraphrase and a
rule that the doc wins on disagreement.

Also: assume JWT is on unless told otherwise (it is the default at key creation
and cannot be changed afterwards), and surface the FCM Data-notifications
dropdown whenever FCM is in play rather than only on from-scratch push setup.

The pitfall index had drifted to 19 entries against 21 headings; it is back at
22 = 22. Version bumped so the changes reach installed users.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An app that already sends its own push (order status, shipping, security
codes) has a FirebaseMessagingService at the default priority 0. Iterable's
registers at -1, so installing the SDK does not steal their dispatch — but
Iterable then receives nothing until their service forwards to it. The
tempting "fix" for that silence is to delete their service, repoint the
manifest entry, or add tools:node="remove", which trades a silent marketing
failure for a broken transactional channel.

The Android corpus documents the forwarding contract as mandatory, but
nothing routed there and neither SKILL.md mentioned coexistence at all. A
non-greenfield eval scenario confirmed both arms got this right from model
competence alone, reverse-engineering the AAR manifest without ever opening
the corpus doc — the same fragility that produced the nav defect.

Android rule 8 + pitfall 23 and the RN equivalents (rule 8, pitfall 14) close
that. The RN side says plainly that its corpus does not cover this, that
Expo must not hand-edit android/, and that iOS coexistence is a different
mechanism. Step 0 now greps for an existing service before scoping push work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PITFALLS #2 said the init callback "should log initialization and nothing
else". The canonical pattern in SKILL.md said "keep it empty" and shipped a
comment-only body. A client trial followed the canonical pattern — correctly,
it is the one labelled "start here" — and read "should log" as optional.

Both halves meant the same thing: do not call SDK methods there. Only one of
them said it in code someone copies, so make that one concrete. The log line
earns its place: the SDK's verbose output proves the SDK initialized, but only
the callback's own line proves the callback ran, which is the difference
between "init never fired" and "init fired, identify didn't".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three gaps a client trial hit in one sitting, all in the "prerequisites you
were handed are correct" assumption.

Pitfall 16 carried only a Groovy snippet, complete with a Groovy scoping note.
Kotlin DSL is the default for new projects and the block does not compile
there: `withInputStream` does not exist, and `java.util.Properties` needs an
explicit import because Kotlin DSL has no auto-imports. The Kotlin form also
needs `buildFeatures { buildConfig = true }`, which the Groovy version never
mentioned because AGP defaulted it on before 8.0. Measured on AGP 8.4: without
it the build fails with "defaultConfig contains custom BuildConfig fields, but
the feature is disabled", which reads like the field is wrong rather than the
flag is off, so the message is quoted.

The trial also found a stale ITERABLE_API_KEY already sitting in
local.properties, from a different project, and caught it only by diffing
against the key it had been given. Nothing told it to look. Same shape for
google-services.json: nothing said to check package_name against applicationId
before building, and the build failure names the file rather than the mismatch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
SKILL.md's done-condition stops at "the SDK is initialized, the user is
identified, the key is wired, the project builds". The push corpus doc ends at
Test Push in the dashboard. A developer who reads both cannot tell which one
they have to satisfy, and the second needs access many app developers don't
have. A client trial spent five minutes deciding it was finished and settled on
a logcat grep it made up.

Four device-side checks instead, all run against a real emulator today. The
arrival check is dumpsys notification rather than a logcat marker on purpose:
the OS is neither the sender nor the app, so it corroborates both, and it needs
no instrumentation in the app under test.

Step 2 exists because the trial hit it: registerDeviceToken answered Success
and getMessages immediately answered UnknownEmailError for the same address.
The profile had simply not propagated. From logcat alone that looks like a
defect, and the repair for a non-defect is a regression. No bound is quoted —
it has been observed once, which is not a measurement.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The canonical pattern takes identity as a String, which assumes it exists at
Application.onCreate(). Real apps often keep it in a DataStore or ViewModel
Flow that emits at login and again on account switch. This was the single
largest cost in a client trial — fifteen minutes to work out that the value has
to be collected and that setEmail still belongs inside onSDKInitialized, with
pitfalls #2 and #3 each pulling in a plausible wrong direction.

Both wrappers are needed and the section says why: onSDKInitialized covers an
identity that arrives before the SDK is ready, the collector covers one that
arrives after. It also names the two shortcuts that look like simplifications —
hoisting setEmail into the init callback, and first()-ing the flow back into a
String, which silently drops the account switch.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A corpus or skill change without a version bump sits on main and no installed
plugin picks it up. One bump for the four commits above rather than one each,
so no intermediate version is published that nobody ever ran.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three claims a reader would have followed to the wrong place:

- The RN "applies to both workflows" label still named pitfall 14, which
  the rebase onto main had turned into an Expo/iOS-only item.
- The Android POST_NOTIFICATIONS note cited pitfall #4 (postDelayed
  sequencing) instead of #7, sending the reader past the permission check
  the note exists to force.
- The definition of done claimed a notification channel observable before
  any push. The SDK creates it while building a notification
  (IterableNotificationHelper, single call site), so it cannot be there
  yet; its name is overridable by manifest meta-data and its id is the
  package name only for default-sound pushes. Dropped the step — the
  push-arrived check that follows it proves more.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A key from a different Iterable project is a valid key: it authenticates,
every request succeeds, and the token, profile and events land in someone
else's project. The bullet named that case but offered a 401 as the
confirmation signal, so an agent that saw clean 200s could read the stale
key as verified and ship it — inverting a "stop and ask" into a "proceed".
Name the real symptom instead: none of it shows up in their dashboard.

Also say where the Kotlin DSL import goes. Imports must be at the top of
the file, but the rest of the block belongs beside the existing
`android { }`, so pasting it whole is a compile error.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- The decision flow said "check rules 1–5" when there are 8. Rule 8 is
  forward-don't-replace, the one that matters most for the step's own
  subject — whatever the user already has — so the flow never reached it.
- "the remaining ~10 traps" is ~15 (23 pitfalls, 8 rules).
- The mobile-key bullet paraphrased a nav path two bullets from the rule
  forbidding that. It is Integrations > API Keys.
- The FCM-type absolute now names its one documented exception: mirroring
  a production integration into a sandbox, where fcm-http-v1-migration
  says match the production app definition. The absolute stays — on
  Notification messages the SDK only logs, so tracking never fires — but
  it no longer contradicts the doc, or this skill's own "the doc wins".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…sible 401

A second review, independent of the first:

- Both citations of "Handling Firebase push messages and tokens" wrote it
  as `##`. It is `####`, nested under `### Step 5`. An agent grepping for
  the level it was given finds nothing and skips the forwarding procedure —
  the exact failure rule 8 and pitfall #23 exist to prevent.
- PITFALLS.md still called SKILL.md's always-on rules "the five". There are
  eight; a reader who counts stops before forward-don't-replace.
- The stale-key bullet promised a 401 "with no obvious cause". The body that
  names it logs at VERBOSE, so at the default log level even the 401
  surfaces as nothing — the same shape as the bug this bullet just fixed for
  the other case.
- The merged-manifest check said to confirm their service is "the priority-0
  entry". The -1 sits on Iterable's intent-filter and their service declares
  no priority at all, so there is no literal 0 to find.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@franco-zalamena-iterable
franco-zalamena-iterable merged commit 9128eff into main Sep 25, 2026
1 check passed
@franco-zalamena-iterable
franco-zalamena-iterable deleted the skill/eval-and-trial-fixes branch September 25, 2026 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant