Skip to content

feat(benchmark): add OpenAI Decisions benchmark providers - #715

Merged
zzwong merged 6 commits into
zzwong/offline-mammoth-identityfrom
zzwong/offline-openai-decisions-identity
Nov 10, 2026
Merged

zzwong merged 6 commits into
zzwong/offline-mammoth-identityfrom
zzwong/offline-openai-decisions-identity

Conversation

@zzwong

@zzwong zzwong commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add the openai_decisions provider type to report-identity suites. Use cr benchmark run identity.yml --provider openai --model gpt-6-luna or select a saved candidate. The configured provider supplies the endpoint and credential store; the adapter handles answer arrays, refusals, bounded responses and cache usage.

The provider work from the System One foundation, Clef, shared execution and credential/replay contract fixes is included. The unshipped infer-openai-decisions command is removed without an alias. Live review behavior is unchanged.

Usage is retained on refused or malformed responses when trustworthy. The two-rate estimate applies only when cache reads and writes are explicitly zero; missing or nonzero cache counts keep cost unknown.

Validation

  • Full tests, lint, module tidy and build passed. Hermetic provider and command regressions cover refusals, malformed answers, cache-write uncertainty, selected-store credentials and complete replayable inputs.
  • Compiled CLI checks cover direct and saved candidates, strict selection errors, empty fixtures, whole-budget rejection, resolved provenance, original retention, independent replay and private artifacts without provider calls.
  • Four prepared-request hashes match the preserved Decisions smoke. That historical transport evidence comprises four successful synthetic comparisons, 1,824 input tokens, zero output tokens and an estimated $0.0001824 of usage value; it used the prior command. No new inference calls or full matrix were run.

Validate named choice answers and cache usage through a dedicated Decisions transport, retaining safe metadata on refusals and failed requests.
Use one versioned report projection and replay runner for System One and Decisions. Preserve request provenance, retained originals and failure accounting while keeping cache-dependent costs explicitly unknown.
@zzwong zzwong changed the title feat(benchmark): add OpenAI Decisions identity inference feat(benchmark): add OpenAI Decisions benchmark providers Nov 7, 2026

@zzwong zzwong left a comment •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a command-level regression for nonzero cache-write costs remaining unknown. The adapter handles this case, but the command test does not exercise it.

Support Clef and Clef Flash through provider-driven identity runs with explicit account credentials and recorded context-completeness limits. Reuse shared execution and preserve complete inputs for offline replay.
@zzwong
zzwong marked this pull request as ready for review November 10, 2026 14:45
@zzwong
zzwong merged commit 38b71ce into zzwong/offline-mammoth-identity Nov 10, 2026
12 checks passed
@zzwong
zzwong deleted the zzwong/offline-openai-decisions-identity branch November 10, 2026 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant