Repository navigation
USAi catalog: unify the refresh lifecycle (single entrypoint, sync-usai-models.mjs as a real emitter, drift-detection CI) #419
Description
Problem
integrations/providers/usai/'s own README documents the intended architecture: catalog.json is the single, harness-neutral source of truth, with per-harness "emitters" (currently opencode.mjs, prime-agent.mjs) rendering it into each harness's own config shape. This is the right design, but it isn't actually wired into a lifecycle — refreshing the full set of downstream artifacts from live USAi data today requires three separate, manually-ordered steps:
node integrations/providers/usai/scripts/build-catalog.mjs --models-url ... --models-dev ...→ regeneratescatalog.json- A manual
node -eone-liner callingemitPrimeAgentModelsFromCatalog→ regenerates theprime-agentkit'smodels.json node integrations/isolation/acq-kits/usai-provider/scripts/sync-usai-models.mjs(and its byte-identical legacy copy insbx-kits/usai-provider-kit/) → regenerates theusai-providerkit'sopencode.jsoncmodel block, via an entirely separate, independent live-data pull — not derived fromcatalog.jsonat all.
This is exactly the setup that let two of these three artifacts silently drift apart from each other and from live USAi data — found on 2026-09-15 while trying to ship a routine catalog refresh (PR #417): catalog.json was stale (still carrying an old 15-model snapshot) while opencode.jsonc had already been regenerated separately, and neither would have been caught before merge if not for a lucky-timing regression test (models-json-regen.test.mjs / emitters.test.mjs — the byte-exact round-trip guards). The live USAi model list itself changes multiple times per day right now (confirmed: went from 21→22 live models within roughly an hour during that same PR's fix cycle), so this isn't a rare edge case — it's the normal operating condition for this catalog.
Proposed work (three pieces, can land as separate PRs)
- A single refresh entrypoint (
npm run refreshinintegrations/providers/usai/, or amake usai-catalog-refreshtarget) that does: live-pull → writecatalog.json→ run every registered emitter → write every downstream artifact, as one atomic operation. This is the direct fix for the drift bug above — no more manually remembering the right order across 3 disconnected scripts in 2 different directories. - Convert
sync-usai-models.mjs(and itssbx-kitscopy) into a real emitter consumingcatalog.json, matching theopencode.mjs/prime-agent.mjspattern, instead of independently re-pulling live data with its own separate model-selection/display-name logic (ROLE_PRIORITY,DISPLAY_NAME_OVERRIDES, etc. currently live only in that script). This removes a real architectural inconsistency the README's own documented data-flow diagram doesn't account for. - A scheduled CI job (weekly cron, matching the existing
test_ruff_pin_lockstep.py-style guard pattern already in this repo) that runs the live pull and either fails a check or opens a tracking issue ifcatalog.jsonhas drifted from live USAi data — turning staleness into something CI catches automatically on a schedule, rather than relying on a human happening to touch the file, or on a coincidental test failure like the one that caught fix(usai-provider): USAi model catalog refresh + NIST-grounded permission hardening #417.
Why this matters beyond tidiness
- Every new harness/agent kit that needs its own model-catalog format (the next candidate is likely
pi-coding-agent, per the deferred emitter noted in patterns#398) will hit the same "which of the 3 scripts do I run, in what order, and how do I know they're all consistent afterward" problem this issue exists to remove. Fixing the lifecycle now, before more emitters exist, is cheaper than fixing it after 4-5 harnesses each have their own ad-hoc regen step. - No documented "add a new harness" procedure exists today. Once (1) and (2) above land, this is also the natural place to write that runbook/template, since a real, working single-entrypoint lifecycle is what such a runbook would actually document.
Priority
Not urgent/blocking — the immediate catalog-staleness bug that surfaced this gap is being fixed directly in PR #417 without this refactor. This issue is the follow-up to prevent the SAME class of bug recurring, and to make onboarding a future harness's emitter straightforward. Recommend medium priority: real risk of recurrence (the live catalog changes multiple times a day, confirmed), but not blocking any current shipped functionality.
Acceptance criteria
- A single command regenerates
catalog.jsonand every registered downstream artifact from one live pull, and is documented as the way to refresh the catalog (README updated to point at it, replacing the current 3-separate-scripts description) -
sync-usai-models.mjs(both copies) consumescatalog.jsonvia the same emitter interface asopencode.mjs/prime-agent.mjs, rather than independently re-pulling live data - A scheduled CI job detects catalog/live-API drift on a regular cadence (weekly suggested) and surfaces it (failing check and/or auto-filed issue) rather than relying on a human or a coincidental test failure to notice
- A short "adding a new harness emitter" doc/runbook exists once the above lands
Activity
mogul commentedon Sep 16, 2026
This would definitely be a good interim improvement, but I'm thinking we will soon want to go further by refactoring per-agent config generation into dependent agent-installation kits, based on the work proposed in GSA-TTS/agentic-coding-quickstart#474.
wz-gsa commentedon Sep 21, 2026
Status note for whoever picks this up next: PR #436 (merged/pending) added
usai-provider/scripts/normalize-catalog.mjs, which readscatalog.jsonand emits a different neutral shape (acq-neutral-model-catalog/v1, for cross-provider discovery per ADR 0003 — #423) than what this issue's item 2 originally asked for (convertingsync-usai-models.mjsinto acatalog.json-consuming emitter matchingopencode.mjs/prime-agent.mjs's existing pattern, to renderopencode.jsoncitself).These are two distinct, non-conflicting needs —
normalize-catalog.mjsdoesn't touchopencode.jsoncrendering or replacesync-usai-models.mjs's current independent live-pull. Flagging so this issue's eventual implementation doesn't collide with or duplicate that work, and so the "adding a new harness emitter" runbook this issue's acceptance criteria calls for accounts for both emitter kinds (harness-config renderers vs. the new provider-discovery normalizer) once both exist.No change to @mogul's deferral call — still correctly gated on GSA-TTS/agentic-coding-quickstart#474 (ADR-0030) landing further, per the comment above.