Skip to content

USAi catalog: unify the refresh lifecycle (single entrypoint, sync-usai-models.mjs as a real emitter, drift-detection CI) #419

Open

Description

@wz-gsa

Problem

integrations/providers/usai/'s own README documents the intended architecture: catalog.json is the single, harness-neutral source of truth, with per-harness "emitters" (currently opencode.mjs, prime-agent.mjs) rendering it into each harness's own config shape. This is the right design, but it isn't actually wired into a lifecycle — refreshing the full set of downstream artifacts from live USAi data today requires three separate, manually-ordered steps:

  1. node integrations/providers/usai/scripts/build-catalog.mjs --models-url ... --models-dev ... → regenerates catalog.json
  2. A manual node -e one-liner calling emitPrimeAgentModelsFromCatalog → regenerates the prime-agent kit's models.json
  3. node integrations/isolation/acq-kits/usai-provider/scripts/sync-usai-models.mjs (and its byte-identical legacy copy in sbx-kits/usai-provider-kit/) → regenerates the usai-provider kit's opencode.jsonc model block, via an entirely separate, independent live-data pull — not derived from catalog.json at all.

This is exactly the setup that let two of these three artifacts silently drift apart from each other and from live USAi data — found on 2026-09-15 while trying to ship a routine catalog refresh (PR #417): catalog.json was stale (still carrying an old 15-model snapshot) while opencode.jsonc had already been regenerated separately, and neither would have been caught before merge if not for a lucky-timing regression test (models-json-regen.test.mjs / emitters.test.mjs — the byte-exact round-trip guards). The live USAi model list itself changes multiple times per day right now (confirmed: went from 21→22 live models within roughly an hour during that same PR's fix cycle), so this isn't a rare edge case — it's the normal operating condition for this catalog.

Proposed work (three pieces, can land as separate PRs)

  1. A single refresh entrypoint (npm run refresh in integrations/providers/usai/, or a make usai-catalog-refresh target) that does: live-pull → write catalog.json → run every registered emitter → write every downstream artifact, as one atomic operation. This is the direct fix for the drift bug above — no more manually remembering the right order across 3 disconnected scripts in 2 different directories.
  2. Convert sync-usai-models.mjs (and its sbx-kits copy) into a real emitter consuming catalog.json, matching the opencode.mjs/prime-agent.mjs pattern, instead of independently re-pulling live data with its own separate model-selection/display-name logic (ROLE_PRIORITY, DISPLAY_NAME_OVERRIDES, etc. currently live only in that script). This removes a real architectural inconsistency the README's own documented data-flow diagram doesn't account for.
  3. A scheduled CI job (weekly cron, matching the existing test_ruff_pin_lockstep.py-style guard pattern already in this repo) that runs the live pull and either fails a check or opens a tracking issue if catalog.json has drifted from live USAi data — turning staleness into something CI catches automatically on a schedule, rather than relying on a human happening to touch the file, or on a coincidental test failure like the one that caught fix(usai-provider): USAi model catalog refresh + NIST-grounded permission hardening #417.

Why this matters beyond tidiness

  • Every new harness/agent kit that needs its own model-catalog format (the next candidate is likely pi-coding-agent, per the deferred emitter noted in patterns#398) will hit the same "which of the 3 scripts do I run, in what order, and how do I know they're all consistent afterward" problem this issue exists to remove. Fixing the lifecycle now, before more emitters exist, is cheaper than fixing it after 4-5 harnesses each have their own ad-hoc regen step.
  • No documented "add a new harness" procedure exists today. Once (1) and (2) above land, this is also the natural place to write that runbook/template, since a real, working single-entrypoint lifecycle is what such a runbook would actually document.

Priority

Not urgent/blocking — the immediate catalog-staleness bug that surfaced this gap is being fixed directly in PR #417 without this refactor. This issue is the follow-up to prevent the SAME class of bug recurring, and to make onboarding a future harness's emitter straightforward. Recommend medium priority: real risk of recurrence (the live catalog changes multiple times a day, confirmed), but not blocking any current shipped functionality.

Acceptance criteria

  • A single command regenerates catalog.json and every registered downstream artifact from one live pull, and is documented as the way to refresh the catalog (README updated to point at it, replacing the current 3-separate-scripts description)
  • sync-usai-models.mjs (both copies) consumes catalog.json via the same emitter interface as opencode.mjs/prime-agent.mjs, rather than independently re-pulling live data
  • A scheduled CI job detects catalog/live-API drift on a regular cadence (weekly suggested) and surfaces it (failing check and/or auto-filed issue) rather than relying on a human or a coincidental test failure to notice
  • A short "adding a new harness emitter" doc/runbook exists once the above lands

Activity

  1. mogul commented on Sep 16, 2026

    @mogul
    Contributor

    This would definitely be a good interim improvement, but I'm thinking we will soon want to go further by refactoring per-agent config generation into dependent agent-installation kits, based on the work proposed in GSA-TTS/agentic-coding-quickstart#474.

  2. wz-gsa commented on Sep 21, 2026

    @wz-gsa
    ContributorAuthor

    Status note for whoever picks this up next: PR #436 (merged/pending) added usai-provider/scripts/normalize-catalog.mjs, which reads catalog.json and emits a different neutral shape (acq-neutral-model-catalog/v1, for cross-provider discovery per ADR 0003 — #423) than what this issue's item 2 originally asked for (converting sync-usai-models.mjs into a catalog.json-consuming emitter matching opencode.mjs/prime-agent.mjs's existing pattern, to render opencode.jsonc itself).

    These are two distinct, non-conflicting needs — normalize-catalog.mjs doesn't touch opencode.jsonc rendering or replace sync-usai-models.mjs's current independent live-pull. Flagging so this issue's eventual implementation doesn't collide with or duplicate that work, and so the "adding a new harness emitter" runbook this issue's acceptance criteria calls for accounts for both emitter kinds (harness-config renderers vs. the new provider-discovery normalizer) once both exist.

    No change to @mogul's deferral call — still correctly gated on GSA-TTS/agentic-coding-quickstart#474 (ADR-0030) landing further, per the comment above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions