Repository navigation
Epic: harness-neutral USAi model catalog + per-harness emitters #357
Description
Problem
The USAi model catalog (models, context windows, prices, backend routing) lives only inside integrations/isolation/acq-kits/usai-provider/'s opencode.jsonc, produced by a ~900-line sync-usai-models.mjs that fetches feeds, enriches, AND renders the OpenCode block in one pass. No other harness (prime-agent, Codex, Claude Code, Copilot CLI) can read it, so every new harness means a second fetch-and-enrich pipeline or hand-transcribed prices that go stale silently.
Approach (verified incremental + additive; 7/7 consensus)
Split into integrations/providers/usai/: live feeds → catalog.json → one small emitter per harness, guarded by a byte-exact round-trip test that regenerates the LIVE committed opencode GENERATED USAI MODELS block and asserts equality against the real shipped file (not a fixture). This test already caught a real tiered-pricing (context_over_200k) bug.
Phase A (add, zero cutover risk): add the catalog + opencode emitter + round-trip test WITHOUT changing the shipped opencode.jsonc. Phase B (later, gated by the green round-trip test): reduce sync-usai-models.mjs to a thin build-catalog + emit wrapper.
MVP — DONE
All five shipped. Measured at origin/main 665373d: node --test gives 63/63 in integrations/providers/usai, and make test-kits runs that suite in CI (ci.yml, after a pinned setup-node).
- Catalog schema +
integrations/providers/usai/scaffold -
build-catalog.mjs(live feeds →catalog.json; bounded fetch, backend-routed models.dev enrichment) - OpenCode emitter + byte-exact round-trip test (the load-bearing guarantee)
- prime-agent emitter + no-secret/shape tests
- Generate + commit the prime-agent
models.json(consumed by the prime-agent kit)
The round-trip guard is real: it reads both shipped opencode.jsonc copies by relative path, so catalog/config drift fails CI rather than a fixture agreeing with itself.
Phase B — what "deliver OpenCode" actually means now
An OpenCode kit already exists twice over — acq-kits/opencode/ installs the pinned, SHA-256-verified binary and declares the agent: block, and acq-kits/usai-provider/ ships a 20-model opencode.jsonc merged into the global config at startup. So shipping OpenCode is not the gap.
The gap is that the pipeline's render half is not wired to anything. emitters/opencode.mjs has zero non-test callers (3 grep hits, all in tests/emitters.test.mjs). The shipped config is still written by sync-usai-models.mjs, which contains zero references to catalog.json and does its own fetch, vendor classification, matching and rendering. The dependency currently runs backwards: catalog.json records "bootstrappedFrom" the shipped opencode.jsonc. The emitter is a verified-correct shadow of the monolith.
Closing that is the cutover below, and it has a prerequisite that was not previously recorded.
Ordered work (OpenCode first)
- Harden the matcher — the models.dev enrichment matcher scored candidates but never rejected one, so a gateway model absent from the enrichment source was enriched with the nearest same-family model's data (measured:
claude_4_5_haikucame out with Opus limits and a 5x price overstatement). PR open: fix(usai-catalog): refuse a near-miss instead of enriching from the wrong model #484. - Validate feed-controlled strings before the emitter becomes load-bearing. A model's
idandnamearrive from the feed verbatim and are interpolated into the generated JSONC with no escaping;catalog.schema.jsondeclares both as bare{"type": "string"}with nopatternand nomaxLength. This is reported privately via the repository's Security tab perSECURITY.md, and is sequenced before the cutover deliberately: the monolith currently self-corrupts on such input (itsnameline is derived from theid, so an escapingidmangles both lines), and that accidental mitigation disappears the moment the emitter — which takesnamefrom an independent field — becomes the writer. - Default-model selection in the catalog pipeline (moved up from "Later" — it is a hard prerequisite, not a nice-to-have). OpenCode's role keys (
model,small_model,agent.*.model,compaction.model) sit at lines 364-386 of the shipped config, outside the emitter's marker block (94-357), andcatalog.jsoncarries no role data at all.sync-usai-models.mjsowns them via aROLE_PRIORITYconstant. So the emitter cannot produce a complete config until role selection has a home, and the cutover cannot proceed without this. Two constraints found while scoping it: nothing currently cross-checks a role id against the model map (a renamed default satisfies both shipped assertions — "key exists" and "starts withusai/" — while matching no model, so it ships dangling and silent), and role vocabulary must not leak OpenCode-specific names (small_model,compaction) into a deliberately harness-neutral catalog. - Reduce
sync-usai-models.mjs→ build-catalog + emitter (the cutover; gated by the round-trip test and by the two items above). Also needs: the two generators currently disagree on model sort order — 17 of 20 positions differ — so running the monolith today reorders the block and breaks the byte-exact test; and the monolith has no post-write validation, so it wrote an unparseable config, printedUpdated … with 1 models, and exited 0.
Later (follow-on)
- Codex / Claude Code / Copilot CLI emitters — needs a research spike first. A repo-wide search found no model/provider configuration for any of the three: no kit, template, ADR, fixture, or
config.toml. The load-bearing safety property here is a byte-exact round-trip against a committed, human-reviewed artifact, and no such artifact can exist for a harness whose config shape is not yet established. The repo already set this precedent —pi-coding-agentdeliberately cut itsmodels.jsonstep rather than ship an unpopulated placeholder with no emitter to fill it. So: spike the target shape, commit a reviewed reference config, then write the emitter against it. Codex first. - prime-agent follow-up (demoted from the MVP's consumer to an optional follow-up — OpenCode is the main path). The emitter→artifact link is real and CI-gated by a byte-exact lockstep test, but the in-guest consumer (
prime-agent-install-models-json.sh) is still a placeholder that onlyechos. Three of the kit's own documents also still assert the emitter "does not exist yet" and thatmodelsis empty — both false since the MVP landed. - Promote
reasoning+inputModalities(read from models.dev today, then discarded; declared in the schema with"Not populated yet.") - Second round-trip test for the sbx-kit
usai-provider-kitcopy — landed with the opencode emitter; the test asserts against both shipped copies. - Reconcile the coexisting catalog schemas (tracked by chore(isolation): formalize JSON Schema for acq-provider-facts/v1 and acq-neutral-model-catalog/v1 #435). There are now four shapes describing one provider:
usai-model-catalog/v1(this epic),acq-neutral-model-catalog/v1(already shipping in the kit asmodel-catalog-snapshot.json),acq-provider-facts/v1(pending, docs(isolation): add ADR 0004 for neutral model-provider discovery #423), and the credential/diagnostic metadata in feat(provider-facts): declare constrained provider credential operations #486. The credential env-var name alone is currently recorded in three of them.
Notes / conditions (from consensus review)
- CI determinism:
build-catalog/ round-trip tests must run against a pinned fixture/snapshot offline (the network live-feed path is a separate--models-url/--no-enrichmentmode) so CI isn't flaky. - §4.3 inspired-not-copied: the scripts/emitters/schema are the repo's own code (may be structurally similar to prior art);
catalog.jsoncontent must be re-derived from live feeds, never copied from any stale snapshot. - Dependency: the MVP was a blocking upstream dependency of the prime-agent kit's
models.json, and that dependency is discharged — the kit ships the real generated artifact today. The remaining prime-agent work is its in-guest install step, not the catalog. - Sequencing rationale: the ordering above was reviewed by a 7-seat panel against an inlined evidence artifact (6 of 7 seats decided, 0 rejections, 5 of 6 selecting this sequence; recorded as
no_quorumunderabsolute_quorumbecause one seat could not read the artifact). The dissent argued for fixing every correctness defect before any structural work — a reasonable position, and the two defects it most cared about (input validation, the dangling-default guard) are inside the sequence above rather than deferred behind it.
Built fresh on GSA-TTS canonical. AI-assisted (OpenCode).
Activity
wz-gsa commentedon Aug 24, 2026
This catalog epic's MVP (prime-agent emitter GSA-TTS/agentic-coding-quickstart#361 → committed models.json GSA-TTS/agentic-coding-quickstart#362) is a blocking upstream dependency of the prime-agent kit epic GSA-TTS/agentic-coding-quickstart#368 (its models.json step #467). Phase A is additive (no change to the shipped opencode.jsonc). Also gated behind the quickstart mawk blocker GSA-TTS/agentic-coding-quickstart#374 for any live kit test. Cross-repo: quickstart epic #467.
feat(usai-catalog): harness-neutral USAi model catalog — schema, buil…
docs(agents): add Durable-References guidance (ephemeral refs vs NIST…
wz-gsa commentedon Aug 27, 2026
Grooming update (2026-08-27): MVP fully shipped and closed — GSA-TTS/agentic-coding-quickstart#358/#359/#360/#361/#362 all verified merged + independently re-tested (byte-exact round-trip + prime-agent shape/no-secret + models.json lockstep guard, 15/15 + 3/3 passing). #367 also closed — its ask (round-trip coverage for both shipped configs) landed as a side effect of GSA-TTS/agentic-coding-quickstart#386's emergency fix. Remaining Later items, all confirmed genuinely incomplete: GSA-TTS/agentic-coding-quickstart#363 (sync-script cutover), GSA-TTS/agentic-coding-quickstart#364 (more emitters), #365 (default-model selection), GSA-TTS/agentic-coding-quickstart#366 (reasoning/inputModalities). Keep epic open for these 4.
wz-gsa commentedon Oct 8, 2026
Rewrote the body to match measured state. Summary of what changed and why, so the diff is reviewable rather than just applied.
The MVP is done — all five boxes ticked
Your 2026-08-27 grooming comment already said so; the body's checkboxes were the only thing left stale. Verified at
origin/main665373d:node --testgives 63/63 inintegrations/providers/usai, andmake test-kitsruns that suite in CI after a pinnedsetup-node. The second sbx-kit round-trip test from the "Later" list is also ticked — it landed with the opencode emitter, which asserts against both shipped copies.One correction to that grooming comment, for anyone reading it later: it cites
GSA-TTS/agentic-coding-quickstart#358-#362for the sub-issues, but those are patterns issues —GSA-TTS/agentic-coding-patterns#358through#362. quickstart#358-#362are unrelated docs PRs. I have not edited your comment; flagging it here so the trail is correct.What "deliver OpenCode" means now
An OpenCode kit already exists twice over —
acq-kits/opencode/installs the pinned binary and declares theagent:block,acq-kits/usai-provider/ships a 20-modelopencode.jsoncmerged at startup. Shipping OpenCode is not the gap.The gap is that the pipeline's render half is wired to nothing:
Zero non-test callers, and the 920-line monolith that actually writes the shipped file never reads
catalog.json. The dependency runs backwards —catalog.jsonrecords"bootstrappedFrom"the shippedopencode.jsonc. So the emitter is a verified-correct shadow of the monolith, and Phase B is the whole remaining OpenCode story.One reordering, with a reason
Default-model selectionmoved from "Later" into the ordered Phase-B work, because it is a hard prerequisite of the cutover rather than an enhancement. This was not previously recorded anywhere:The role keys sit outside the emitter's marker block, and
catalog.json's top-level keys are onlyschemaVersion, generatedBy, sources, gateway, vendors, models— no role data at all. The monolith owns them via aROLE_PRIORITYconstant. The emitter therefore cannot produce a complete config until role selection has a home, so the cutover is blocked on it.Two things found while scoping that item:
usai/". I renamedsmall_modeltousai/claude_4_5_haiku_RENAMED_AWAY— it satisfies both while matching no model in the block. A refresh that renames a model ships a dangling default silently.small_modelandcompactionare OpenCode's names. Putting them incatalog.jsonwould couple a deliberately harness-neutral artifact to one harness — which is the problem this epic exists to solve. Worth deciding the shape before implementing.Codex: a spike, not a fast-follow
I wanted Codex to be a quick next step and it cannot be, for a reason worth recording:
There is no model/provider configuration for Codex, Claude Code or Copilot anywhere in the repo — no kit, template, ADR, or fixture. The only hits for "codex" are rules-file path conventions, egress hostnames, and a model SKU name. Since the load-bearing safety property here is a byte-exact round-trip against a committed, human-reviewed artifact, there is nothing for a Codex emitter to be correct against. Writing one today means guessing at a format and then testing the guess against itself.
The repo already set the precedent —
pi-coding-agentdeliberately cut itsmodels.jsonstep rather than ship an unpopulated placeholder with no emitter to fill it. So Codex is: spike the shape → commit a reviewed reference config → write the emitter. Codex first among the three.prime-agent demoted
Its emitter→artifact link is real and CI-gated by a byte-exact lockstep test, so the catalog half is genuinely done — the dependency this epic's MVP was blocking is discharged. What remains is the in-guest install step, which is still a placeholder that only
echos. That makes it an optional follow-up rather than the MVP's consumer, which matches OpenCode being the main path.Separately: three of that kit's own documents still assert the emitter "does not exist yet" and that
modelsis empty. Both are false now. I will file that separately rather than bury it here.Also folded in
Three defects that Phase B has to deal with regardless of ordering:
sync:usai-modelstoday reorders the whole block and breaks the byte-exact test. Partly covered by USAi catalog: unify the refresh lifecycle (single entrypoint, sync-usai-models.mjs as a real emitter, drift-detection CI) #419, but the reorder itself is not stated there.opencode.jsonc, printedUpdated … with 1 models, and exited 0.model-catalog-snapshot.jsonalready shipsacq-neutral-model-catalog/v1in the kit, and nothing in this epic tracked it. Withacq-provider-facts/v1(docs(isolation): add ADR 0004 for neutral model-provider discovery #423) and the credential metadata in feat(provider-facts): declare constrained provider credential operations #486, that is four shapes for one provider — and the credential env-var name is recorded in three of them. Noted against the reconciliation item.Sequencing provenance
The ordering was reviewed by a 7-seat panel against an inlined evidence artifact: 6 of 7 seats decided, 0 rejections, 5 of 6 selecting this sequence. It is recorded as
no_quorumunderabsolute_quorumbecause one seat could not read the artifact — so this is a strong signal, not a ratified verdict, and I would not describe it as approved.The dissent argued for fixing every correctness defect before any structural work. That is a defensible position, and the two defects it cared about most — input validation and the dangling-default guard — are inside the sequence above rather than deferred behind it.
AI-assisted (OpenCode). Advisory — requires human review.