Repository navigation
Conversation
Carries the parser cases and the eval case of #21, anonymized: the #21 draft crossed with typographic, ASCII and code-like characters, plus the past JSON failures, as data files the format evals read. Every response is recorded raw for later scoring (20261031-141038-d-tac-p8l). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pkg/llm.Request gains ResponseSchema. The claude CLI adapter passes it to --json-schema and returns the validated structured_output; the gollm adapter hands it to the fork's WithResponseSchema, which keeps the system prompt split and usage. llm.endpoint now also sets the anthropic base URL. Wire tests cover the schema slot, the prompt split, usage and problem characters both ways (20261031-141038-d-tac-p8l). go.mod points at the local gollm checkout until its commits are pushed and pinned. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pre-flight and the writing guide send their result schema on the request, kept as JSON files so the reasoning-first property order survives; the guide's closed axis and repair sets derive from it. The shared escaping partial and the output-format instructions are gone, the prompts describe what each field means, and a quote or cited excerpt only has to locate its passage, with an example in the guide (20261004-135748-d-tac-ndh, 20261005-170145-d-tac-gch). Format eval rows record the structure mode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guide may shorten a passage or change its quotation marks, so the agent finds the passage from the quote instead of matching it (20261005-170145-d-tac-gch). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Never on push and never for a fork, since the job holds OPPER_API_KEY. It covers the OpenAI adapter with a GPT and a Claude model on Opper's OpenAI-compatible route and the Anthropic adapter on its Messages route, and keeps the raw responses as artifacts for later scoring (20261031-141038-d-tac-p8l). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the temporary local replace with the pushed generate-with-usage commit carrying the schema path, the single per-call system prompt, the OpenAI parameter defaults and the Anthropic base URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SDD-Mutation: v1:b759005b4ae07fd91ddccc30024f5f81f44d2d7a1a06132e57a330a31b198a4b
SDD-Mutation: v1:6f9875db1afc4ed5aa5a0d60991146d2f43c4d93225418014891121bfb4a5fbe
SDD-Mutation: v1:d2c97199f1467897bb55d7f8b08407574b4514537885621e539d31128702bc03
SDD-Mutation: v1:e45c3aac286145ac9f5207c985dd49ab743ea94d2092d88c1b32b49e97fe50a0
Pins the gollm fork at 698361b, which reads usage.cost, and maps it to llm.Usage.CostUSD; Opper reports it with every response (20261101-010604-d-tac-769). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each response row carries tokens and the reported cost; the run ends with validity and cost per provider, model and setting, printed and written as format-summary.md for the workflow's job summary (20261101-010604-d-tac-769). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One path per model at its production setting through Opper, with validity and cost in the job summary (20261101-010604-d-tac-769). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SDD-Mutation: v1:92b9803d46483a43be95c090686c7a1e42628a496c5fa1a0132747c631685d60
SDD-Mutation: v1:17b6257dc538c3e732fc24a803cbb58879e6e0eae309a799fced5a845c10f4cd
SDD-Mutation: v1:ce3f11b413e8f1629d83bb56fe184d4cc4b6b12c31951ead63f87f72e85d123d
SDD-Mutation: v1:1c0353ca2337c2756edddd4f6e7ef45672286d99e05f1bc2da103e8fa8659986
SDD-Mutation: v1:13d30622446d58c9b15f828be2d5e16a686fd0a57d31d552e120095ea5729e2e
Specimens now assert the expected finding and a quote that locates its passage; the synthetic character variants and the validity-only format tests are removed. The evals workflow runs the pre-flight, writing-guide and summarize evals per model through Opper, with summarize's judge on Opper (SDD_EVAL_JUDGE_ENDPOINT), and reports each case and the cost. Refs: 20261102-101250-d-tac-s98 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hlubek
added
eval
glm cites pre-flight excerpts in '...', which the specimen check skipped, so a correct high finding failed the case. A straight single quote now opens an excerpt beside a non-letter, so apostrophes still do not. Refs: 20261102-101250-d-tac-s98 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hlubek
added
eval
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why. Pre-flight and the writing guide asked the model to write JSON as free text. Drafts quoted with German marks („…“) made it emit an unescaped
", the fourth invalid-JSON failure after three fixes at the trigger (#20). JSON validity now becomes the provider's job.What.
pkg/llm.Request, and every adapter enforces it natively: Claude CLI--json-schema, OpenAI strictresponse_format, Anthropicoutput_config.format, Ollamaformat. An endpoint that ignores the schema fails parsing loudly.698361b, carries the schema path with the system prompt and usage intact, sends the system prompt once and per call, picks OpenAI parameters without matching model names, takes a configurable Anthropic base URL and reports Opper's cost.evalsworkflow runs the pre-flight, writing-guide and summarize evals for glm-5.3-flash, gpt-5.6-luna and claude-haiku-5-5 through Opper when a PR gets theevallabel, and reports each case's result and the cost per model.Result. After the change every run returned valid JSON on claude-sonnet-4-6 (37/60 before), gpt-5-mini, the Claude CLI and glm-5.3-flash through Opper. Ollama Cloud ignores schemas.
SDD entries (
sdd show <id>):20261031-141038-d-tac-p8l: the plan, with its alignment record20261004-134853-d-cpt-sw8,20261004-135748-d-tac-ndh: structured output as a guarantee of the LLM boundary, enforced natively only20261005-170145-d-tac-gch: quotes locate the passage, not reproduce it20261101-010604-d-tac-769,20261102-101250-d-tac-s98: the eval workflow's models, and its content checks20261101-002900-s-tac-3q0: Ollama Cloud ignores schemasThe branch lands by local merge; this PR runs the workflow.
🤖 Generated with Claude Code