Skip to content

Epic: harness-neutral USAi model catalog + per-harness emitters #357

Open
0 of 10 issues completed
Open

Description

@wz-gsa

Problem

The USAi model catalog (models, context windows, prices, backend routing) lives only inside integrations/isolation/acq-kits/usai-provider/'s opencode.jsonc, produced by a ~900-line sync-usai-models.mjs that fetches feeds, enriches, AND renders the OpenCode block in one pass. No other harness (prime-agent, Codex, Claude Code, Copilot CLI) can read it, so every new harness means a second fetch-and-enrich pipeline or hand-transcribed prices that go stale silently.

Approach (verified incremental + additive; 7/7 consensus)

Split into integrations/providers/usai/: live feeds → catalog.json → one small emitter per harness, guarded by a byte-exact round-trip test that regenerates the LIVE committed opencode GENERATED USAI MODELS block and asserts equality against the real shipped file (not a fixture). This test already caught a real tiered-pricing (context_over_200k) bug.

Phase A (add, zero cutover risk): add the catalog + opencode emitter + round-trip test WITHOUT changing the shipped opencode.jsonc. Phase B (later, gated by the green round-trip test): reduce sync-usai-models.mjs to a thin build-catalog + emit wrapper.

MVP — DONE

All five shipped. Measured at origin/main 665373d: node --test gives 63/63 in integrations/providers/usai, and make test-kits runs that suite in CI (ci.yml, after a pinned setup-node).

  • Catalog schema + integrations/providers/usai/ scaffold
  • build-catalog.mjs (live feeds → catalog.json; bounded fetch, backend-routed models.dev enrichment)
  • OpenCode emitter + byte-exact round-trip test (the load-bearing guarantee)
  • prime-agent emitter + no-secret/shape tests
  • Generate + commit the prime-agent models.json (consumed by the prime-agent kit)

The round-trip guard is real: it reads both shipped opencode.jsonc copies by relative path, so catalog/config drift fails CI rather than a fixture agreeing with itself.

Phase B — what "deliver OpenCode" actually means now

An OpenCode kit already exists twice over — acq-kits/opencode/ installs the pinned, SHA-256-verified binary and declares the agent: block, and acq-kits/usai-provider/ ships a 20-model opencode.jsonc merged into the global config at startup. So shipping OpenCode is not the gap.

The gap is that the pipeline's render half is not wired to anything. emitters/opencode.mjs has zero non-test callers (3 grep hits, all in tests/emitters.test.mjs). The shipped config is still written by sync-usai-models.mjs, which contains zero references to catalog.json and does its own fetch, vendor classification, matching and rendering. The dependency currently runs backwards: catalog.json records "bootstrappedFrom" the shipped opencode.jsonc. The emitter is a verified-correct shadow of the monolith.

Closing that is the cutover below, and it has a prerequisite that was not previously recorded.

Ordered work (OpenCode first)

  • Harden the matcher — the models.dev enrichment matcher scored candidates but never rejected one, so a gateway model absent from the enrichment source was enriched with the nearest same-family model's data (measured: claude_4_5_haiku came out with Opus limits and a 5x price overstatement). PR open: fix(usai-catalog): refuse a near-miss instead of enriching from the wrong model #484.
  • Validate feed-controlled strings before the emitter becomes load-bearing. A model's id and name arrive from the feed verbatim and are interpolated into the generated JSONC with no escaping; catalog.schema.json declares both as bare {"type": "string"} with no pattern and no maxLength. This is reported privately via the repository's Security tab per SECURITY.md, and is sequenced before the cutover deliberately: the monolith currently self-corrupts on such input (its name line is derived from the id, so an escaping id mangles both lines), and that accidental mitigation disappears the moment the emitter — which takes name from an independent field — becomes the writer.
  • Default-model selection in the catalog pipeline (moved up from "Later" — it is a hard prerequisite, not a nice-to-have). OpenCode's role keys (model, small_model, agent.*.model, compaction.model) sit at lines 364-386 of the shipped config, outside the emitter's marker block (94-357), and catalog.json carries no role data at all. sync-usai-models.mjs owns them via a ROLE_PRIORITY constant. So the emitter cannot produce a complete config until role selection has a home, and the cutover cannot proceed without this. Two constraints found while scoping it: nothing currently cross-checks a role id against the model map (a renamed default satisfies both shipped assertions — "key exists" and "starts with usai/" — while matching no model, so it ships dangling and silent), and role vocabulary must not leak OpenCode-specific names (small_model, compaction) into a deliberately harness-neutral catalog.
  • Reduce sync-usai-models.mjs → build-catalog + emitter (the cutover; gated by the round-trip test and by the two items above). Also needs: the two generators currently disagree on model sort order — 17 of 20 positions differ — so running the monolith today reorders the block and breaks the byte-exact test; and the monolith has no post-write validation, so it wrote an unparseable config, printed Updated … with 1 models, and exited 0.

Later (follow-on)

  • Codex / Claude Code / Copilot CLI emitters — needs a research spike first. A repo-wide search found no model/provider configuration for any of the three: no kit, template, ADR, fixture, or config.toml. The load-bearing safety property here is a byte-exact round-trip against a committed, human-reviewed artifact, and no such artifact can exist for a harness whose config shape is not yet established. The repo already set this precedent — pi-coding-agent deliberately cut its models.json step rather than ship an unpopulated placeholder with no emitter to fill it. So: spike the target shape, commit a reviewed reference config, then write the emitter against it. Codex first.
  • prime-agent follow-up (demoted from the MVP's consumer to an optional follow-up — OpenCode is the main path). The emitter→artifact link is real and CI-gated by a byte-exact lockstep test, but the in-guest consumer (prime-agent-install-models-json.sh) is still a placeholder that only echos. Three of the kit's own documents also still assert the emitter "does not exist yet" and that models is empty — both false since the MVP landed.
  • Promote reasoning + inputModalities (read from models.dev today, then discarded; declared in the schema with "Not populated yet.")
  • Second round-trip test for the sbx-kit usai-provider-kit copy — landed with the opencode emitter; the test asserts against both shipped copies.
  • Reconcile the coexisting catalog schemas (tracked by chore(isolation): formalize JSON Schema for acq-provider-facts/v1 and acq-neutral-model-catalog/v1 #435). There are now four shapes describing one provider: usai-model-catalog/v1 (this epic), acq-neutral-model-catalog/v1 (already shipping in the kit as model-catalog-snapshot.json), acq-provider-facts/v1 (pending, docs(isolation): add ADR 0004 for neutral model-provider discovery #423), and the credential/diagnostic metadata in feat(provider-facts): declare constrained provider credential operations #486. The credential env-var name alone is currently recorded in three of them.

Notes / conditions (from consensus review)

  • CI determinism: build-catalog / round-trip tests must run against a pinned fixture/snapshot offline (the network live-feed path is a separate --models-url/--no-enrichment mode) so CI isn't flaky.
  • §4.3 inspired-not-copied: the scripts/emitters/schema are the repo's own code (may be structurally similar to prior art); catalog.json content must be re-derived from live feeds, never copied from any stale snapshot.
  • Dependency: the MVP was a blocking upstream dependency of the prime-agent kit's models.json, and that dependency is discharged — the kit ships the real generated artifact today. The remaining prime-agent work is its in-guest install step, not the catalog.
  • Sequencing rationale: the ordering above was reviewed by a 7-seat panel against an inlined evidence artifact (6 of 7 seats decided, 0 rejections, 5 of 6 selecting this sequence; recorded as no_quorum under absolute_quorum because one seat could not read the artifact). The dissent argued for fixing every correctness defect before any structural work — a reasonable position, and the two defects it most cared about (input validation, the dangling-default guard) are inside the sequence above rather than deferred behind it.

Built fresh on GSA-TTS canonical. AI-assisted (OpenCode).

Activity

  1. added
    enhancementNew feature or request
    epicParent tracking issue
    validationSchema, linting, tests, CI validation
    on Aug 24, 2026
  2. wz-gsa commented on Aug 24, 2026

    @wz-gsa
    ContributorAuthor

    This catalog epic's MVP (prime-agent emitter GSA-TTS/agentic-coding-quickstart#361 → committed models.json GSA-TTS/agentic-coding-quickstart#362) is a blocking upstream dependency of the prime-agent kit epic GSA-TTS/agentic-coding-quickstart#368 (its models.json step #467). Phase A is additive (no change to the shipped opencode.jsonc). Also gated behind the quickstart mawk blocker GSA-TTS/agentic-coding-quickstart#374 for any live kit test. Cross-repo: quickstart epic #467.

  3. wz-gsa commented on Aug 27, 2026

    @wz-gsa
    ContributorAuthor

    Grooming update (2026-08-27): MVP fully shipped and closed — GSA-TTS/agentic-coding-quickstart#358/#359/#360/#361/#362 all verified merged + independently re-tested (byte-exact round-trip + prime-agent shape/no-secret + models.json lockstep guard, 15/15 + 3/3 passing). #367 also closed — its ask (round-trip coverage for both shipped configs) landed as a side effect of GSA-TTS/agentic-coding-quickstart#386's emergency fix. Remaining Later items, all confirmed genuinely incomplete: GSA-TTS/agentic-coding-quickstart#363 (sync-script cutover), GSA-TTS/agentic-coding-quickstart#364 (more emitters), #365 (default-model selection), GSA-TTS/agentic-coding-quickstart#366 (reasoning/inputModalities). Keep epic open for these 4.

  4. wz-gsa commented on Oct 8, 2026

    @wz-gsa
    ContributorAuthor

    Rewrote the body to match measured state. Summary of what changed and why, so the diff is reviewable rather than just applied.

    The MVP is done — all five boxes ticked

    Your 2026-08-27 grooming comment already said so; the body's checkboxes were the only thing left stale. Verified at origin/main 665373d: node --test gives 63/63 in integrations/providers/usai, and make test-kits runs that suite in CI after a pinned setup-node. The second sbx-kit round-trip test from the "Later" list is also ticked — it landed with the opencode emitter, which asserts against both shipped copies.

    One correction to that grooming comment, for anyone reading it later: it cites GSA-TTS/agentic-coding-quickstart#358-#362 for the sub-issues, but those are patterns issues — GSA-TTS/agentic-coding-patterns#358 through #362. quickstart #358-#362 are unrelated docs PRs. I have not edited your comment; flagging it here so the trail is correct.

    What "deliver OpenCode" means now

    An OpenCode kit already exists twice over — acq-kits/opencode/ installs the pinned binary and declares the agent: block, acq-kits/usai-provider/ ships a 20-model opencode.jsonc merged at startup. Shipping OpenCode is not the gap.

    The gap is that the pipeline's render half is wired to nothing:

    $ grep -rn "emitOpenCodeBlockFromCatalog" --include="*.mjs" --include="*.json" --include="*.yml" --include=Makefile .
    tests/emitters.test.mjs:23  (import)
    tests/emitters.test.mjs:85  (test name)
    tests/emitters.test.mjs:90  (call)
    emitters/opencode.mjs:110   (definition)
    emitters/opencode.mjs:153   (default export)
    
    $ grep -c "catalog.json" acq-kits/usai-provider/scripts/sync-usai-models.mjs
    0
    

    Zero non-test callers, and the 920-line monolith that actually writes the shipped file never reads catalog.json. The dependency runs backwards — catalog.json records "bootstrappedFrom" the shipped opencode.jsonc. So the emitter is a verified-correct shadow of the monolith, and Phase B is the whole remaining OpenCode story.

    One reordering, with a reason

    Default-model selection moved from "Later" into the ordered Phase-B work, because it is a hard prerequisite of the cutover rather than an enhancement. This was not previously recorded anywhere:

    $ grep -nE 'BEGIN GENERATED|END GENERATED|"small_model":|"compaction"' .../opencode.jsonc
     94:  // BEGIN GENERATED USAI MODELS
    357:  // END GENERATED USAI MODELS
    364:  "model": "usai/claude-sonnet-5",
    365:  "small_model": "usai/claude_4_5_haiku",
    370:  "compaction": { ...
    

    The role keys sit outside the emitter's marker block, and catalog.json's top-level keys are only schemaVersion, generatedBy, sources, gateway, vendors, models — no role data at all. The monolith owns them via a ROLE_PRIORITY constant. The emitter therefore cannot produce a complete config until role selection has a home, so the cutover is blocked on it.

    Two things found while scoping that item:

    • Nothing cross-checks a role id against the model map. The shipped assertions are "key exists" and "value starts with usai/". I renamed small_model to usai/claude_4_5_haiku_RENAMED_AWAY — it satisfies both while matching no model in the block. A refresh that renames a model ships a dangling default silently.
    • Role vocabulary should not leak into the neutral catalog. small_model and compaction are OpenCode's names. Putting them in catalog.json would couple a deliberately harness-neutral artifact to one harness — which is the problem this epic exists to solve. Worth deciding the shape before implementing.

    Codex: a spike, not a fast-follow

    I wanted Codex to be a quick next step and it cannot be, for a reason worth recording:

    $ find . -name 'config.toml' -not -path './.git/*'
    (no results)
    

    There is no model/provider configuration for Codex, Claude Code or Copilot anywhere in the repo — no kit, template, ADR, or fixture. The only hits for "codex" are rules-file path conventions, egress hostnames, and a model SKU name. Since the load-bearing safety property here is a byte-exact round-trip against a committed, human-reviewed artifact, there is nothing for a Codex emitter to be correct against. Writing one today means guessing at a format and then testing the guess against itself.

    The repo already set the precedent — pi-coding-agent deliberately cut its models.json step rather than ship an unpopulated placeholder with no emitter to fill it. So Codex is: spike the shape → commit a reviewed reference config → write the emitter. Codex first among the three.

    prime-agent demoted

    Its emitter→artifact link is real and CI-gated by a byte-exact lockstep test, so the catalog half is genuinely done — the dependency this epic's MVP was blocking is discharged. What remains is the in-guest install step, which is still a placeholder that only echos. That makes it an optional follow-up rather than the MVP's consumer, which matches OpenCode being the main path.

    Separately: three of that kit's own documents still assert the emitter "does not exist yet" and that models is empty. Both are false now. I will file that separately rather than bury it here.

    Also folded in

    Three defects that Phase B has to deal with regardless of ordering:

    Sequencing provenance

    The ordering was reviewed by a 7-seat panel against an inlined evidence artifact: 6 of 7 seats decided, 0 rejections, 5 of 6 selecting this sequence. It is recorded as no_quorum under absolute_quorum because one seat could not read the artifact — so this is a strong signal, not a ratified verdict, and I would not describe it as approved.

    The dissent argued for fixing every correctness defect before any structural work. That is a defensible position, and the two defects it cared about most — input validation and the dangling-default guard — are inside the sequence above rather than deferred behind it.

    AI-assisted (OpenCode). Advisory — requires human review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestepicParent tracking issuevalidationSchema, linting, tests, CI validation

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions