diff --git a/README.md b/README.md index 1028a651..26907d81 100644 --- a/README.md +++ b/README.md @@ -133,7 +133,7 @@ $ quest --root examples/export unblocked 1 quest/m0/export.md $ quest --root examples/export unblocked --json --limit 1 -[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"models":[{"harness":"claude","model":"claude-opus-5-5","effort":"low"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}] +[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"models":[{"harness":"claude","model":"claude-opus-5-5","effort":"medium"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}] ``` Each plain output line contains the count, a tab, and the path. On a terminal, @@ -1346,7 +1346,7 @@ the same for every repository unless its `AGENTS.md` names others: | Size | Writer | Fallback | |---|---|---| | XS | Claude Code `claude-sonnet-5-5` high | Codex `gpt-6.1-sol` low | -| S | Claude Code `claude-opus-5-5` low | Codex `gpt-6.1-sol` medium | +| S | Claude Code `claude-opus-5-5` medium | Codex `gpt-6.1-sol` medium | | M | Claude Code `claude-opus-5-5` medium | Codex `gpt-6.1-sol` high | | L | Claude Code `claude-opus-5-5` high | Codex `gpt-6.1-sol` xhigh | | XL | Claude Code `claude-opus-5-5` xhigh | none | @@ -1371,14 +1371,14 @@ claude claude-sonnet-5-5 high codex gpt-6.1-sol low $ quest models XS --writer claude-opus-5-5 -claude claude-opus-5-5 low +claude claude-opus-5-5 medium ``` `--writer` prints the size's entry for that model, or else its entry at the nearest size that lists it, the larger on a tie; a model no list names is an error. `--json` prints an array of `{harness, model, effort}` objects. The iterate records the size it was given in its `Implementer:` line, as -`Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS)`, so +`Implementer: Claude Code, claude-opus-5-5, medium effort (iterate, XS)`, so the model check can audit the routing. `quest review` still reviews the whole branch at the PR's size. @@ -1399,8 +1399,9 @@ size. Each size ranks its reviewers, the same for every repository: Beside it, the shadow, the pick's model one effort lower (Sol low for an XS PR), reviews the same head in its own session, so the routing can be retuned from what each effort step finds over the one below it. It is skipped when the -pick runs at low or at an `--effort` off the ladder (low, medium, high, xhigh, -max). It runs on the branch's first run of the pick, then reruns only while it has findings to settle, or once the pick +pick runs at low or at an `--effort` off the ladder (low, medium, high, xhigh). + +It runs on the branch's first run of the pick, then reruns only while it has findings to settle, or once the pick changed so that it is no longer one step below; its findings are numbered, answered, and settled like any reviewer's. A pick with no step below it seats a shadow at its own effort while the old shadow still has findings, so none @@ -1683,6 +1684,10 @@ quest agent stop --all quest agent admit ``` +`--effort` takes any level the CLI does except `max`, which costs about as much +as a stronger model at a lower effort without matching it; `quest review` +refuses it too. + Codex starts and resumes only in a checkout registered by `quest worktree create`, on Linux with the `quest worktree run` supervisor available. Ownership and host support are checked before any agent state is written. Each turn uses diff --git a/quest/m0/routing-evidence/writer-ab.md b/quest/m0/routing-evidence/writer-ab.md index 11ab9623..8be35ef2 100644 --- a/quest/m0/routing-evidence/writer-ab.md +++ b/quest/m0/routing-evidence/writer-ab.md @@ -1,10 +1,10 @@ -# [M] Split XS, S and M writers between two arms +# [M] Split each size's writers between two arms ## Goal -`quest unblocked --json` routes each XS quest to Sonnet 5.5 high or Opus -low, each S quest to Opus low or medium, and each M quest to Opus medium -or high, 50/50. The split is stable per quest, +`quest unblocked --json` routes each XS and S quest to Sonnet 5.5 high or +Opus 5.5 medium, each M quest to Opus medium or high, and each XL quest to +Opus xhigh or Fable 5.2 medium, 50/50. The split is stable per quest, and the review efforts in `src/cli/models.ts` stay fixed while it runs. Each `Implementer:` line records its arm, iterates stay on the PR's arm, and `quest usage` reports cost, rounds, findings, hand-backs, and later @@ -37,6 +37,26 @@ Decided 2026-11-11 (user, /quest-plan, - Rejected: Sonnet medium against Sonnet high, since the same cost guide puts Sonnet medium about 7 points below high on its SWE-bench Pro subset and 15 below on its internal agentic-coding set; no XS arm +- Re-armed 2026-11-13 (user), to test where Anthropic's + [model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/) + and Theo Browne's + [Opus against Fable video](https://www.youtube.com/watch?v=XtykeVoSmP8) + disagree: + - S Sonnet high against Opus medium: the guide puts scoped coding on + Sonnet, Theo finds Sonnet runs up output tokens and costs about what + Opus does. S's table entry moved from Opus low to Opus medium, the + guide's default, since both call Opus low too weak. + - XS Sonnet high against Opus medium rather than Opus low, the same + contrast as S (agent's choice, since no table entry runs Opus low any + more). + - M Opus medium against high, as before: Theo finds medium ends its turn + too early and calls high the best effort. + - XL Opus xhigh against Fable medium: the guide moves up to Fable rather + than past high, while Theo finds Opus checks verifiable work more + carefully and a subscription gives about a quarter as much Fable. XL's + table entry stays Opus xhigh for that subscription cap (rejected: + Fable medium; dropping the Fable arm). XL quests are rare, so report + how many each arm got rather than wait for a detectable difference. - Escapes are too rare to separate the arms, about 2% of PRs. Compare first-round findings under the fixed review efforts, rounds, hand-backs, and cost, and report escapes alongside. diff --git a/quest/m1/model-check.md b/quest/m1/model-check.md index 614b92e9..aa234994 100644 --- a/quest/m1/model-check.md +++ b/quest/m1/model-check.md @@ -111,6 +111,21 @@ checkpoints and outages to count by hand: puts Sonnet 5.5 high level with Opus 5.5 low (87.6% against 87.4%) at about the same cost per solved task, and medium about 7 points below high on its SWE-bench Pro subset. + - Changed again (maintainer, 2026-11-13), on Anthropic's + [model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/), + which puts scoped coding on Sonnet 5.5 and advises moving up a model + rather than past `high`: S writes on Opus 5.5 medium instead of low, + whose S PRs averaged 4.7 review rounds over 6 reviewed; and nothing + runs `max`, which `quest agent` and `quest review` refuse. XL stays on + Opus xhigh: the maintainer first moved it to Fable 5.2 medium, then + kept Opus, since a Max or Ultra plan caps Fable at half its weekly + limit and Fable uses that limit faster (Theo Browne's + [video](https://www.youtube.com/watch?v=XtykeVoSmP8) estimates about + $3k a month of Fable against $14k of Opus on the $200 plan). XS stays Sonnet high, since `quest usage` had no + Sonnet medium runs to compare. The + [writer A/B](/quest/m0/routing-evidence/writer-ab.md) tests each change + against its alternative, Fable medium on XL included; tally the S cell + fresh. - Compare both tables' models (in `src/cli/models.ts`) with each vendor's current models: the newest model of each tier the tables pin. Bump any that are stale, and check that both CLIs accept the new ids and the pinned efforts. `--deep` and its strongest-model diff --git a/src/cli/main.ts b/src/cli/main.ts index 743bbe23..0fb57672 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -1048,7 +1048,7 @@ export function main(argv: string[], out: Output): number | Promise { command .addOption(new Option("--harness ", "Agent CLI to run").choices(HARNESSES).makeOptionMandatory()) .requiredOption("--model ", "Model the agent runs") - .requiredOption("--effort ", "Reasoning effort the agent runs at") + .requiredOption("--effort ", "Reasoning effort the agent runs at, never max", effortLevel) .requiredOption( "--permission-mode ", "Claude Code's --permission-mode, or Codex's sandbox mode; the agent cannot prompt", @@ -1231,7 +1231,7 @@ export function main(argv: string[], out: Output): number | Promise { "Review the checkout's committed branch against its base with a reviewer its size picks, never the " + `model that changed it last. Each size ranks its reviewers (harness, model, effort): ${reviewers}. The ` + "first whose model is not avoided reviews, and beside it, on the same head in its own session, the shadow: " + - "the pick's model one effort lower (low, medium, high, xhigh, max), skipped when the pick runs at low or an " + + "the pick's model one effort lower (low, medium, high, xhigh), skipped when the pick runs at low or an " + "effort off that ladder, so each effort step can be compared with the one below it. The shadow runs on a " + "branch's first run of the pick, then only while it has findings to settle or once the pick changed so it is " + "no longer one step below; its findings are numbered and settled like any reviewer's, its session carries " + @@ -1317,7 +1317,11 @@ export function main(argv: string[], out: Output): number | Promise { ).choices(HARNESSES), ) .option("--model ", "Model the size's pick, or --harness's reviewer, runs instead") - .option("--effort ", "Reasoning effort the size's pick, or --harness's reviewer, runs at instead") + .option( + "--effort ", + "Reasoning effort the size's pick, or --harness's reviewer, runs at instead, never max", + effortLevel, + ) .option( "--exclude-model ", "A model never picked to review it, instead of the PR's latest writer; repeat for each", @@ -1637,6 +1641,15 @@ export function main(argv: string[], out: Output): number | Promise { } } +/** + * An agent's effort: any level but `max`, which costs as much as a stronger + * model at a lower effort without matching it. + */ +function effortLevel(value: string): string { + if (value === "max") throw new InvalidArgumentError("max is never run; use a stronger model instead"); + return nonEmpty(value); +} + /** An empty path names no directory, so it is a usage error rather than `.`. */ function nonEmpty(value: string): string { if (value === "") throw new InvalidArgumentError("a value is required"); diff --git a/src/cli/models.ts b/src/cli/models.ts index a70456cd..7e6d0a23 100644 --- a/src/cli/models.ts +++ b/src/cli/models.ts @@ -25,11 +25,13 @@ const sonnet = (effort: string): Model => ({ harness: "claude", model: "claude-s * writes a PR runs all its iterates. XS runs Sonnet rather than the cheaper * Haiku, whose surcharge past 100k input tokens leads users to cap its * compaction window there, barely above an agent's fixed load of prompt, - * context, and skill. + * context, and skill. No size runs Opus at low, which misses too much. XL + * stays on Opus rather than Fable, which a subscription caps at half its + * weekly limit and which uses that limit faster. */ export const WRITERS: Record = { XS: [sonnet("high"), sol("low")], - S: [opus("low"), sol("medium")], + S: [opus("medium"), sol("medium")], M: [opus("medium"), sol("high")], L: [opus("high"), sol("xhigh")], XL: [opus("xhigh")], diff --git a/src/cli/review.ts b/src/cli/review.ts index 0ab88765..866c29d8 100644 --- a/src/cli/review.ts +++ b/src/cli/review.ts @@ -1229,8 +1229,8 @@ const seatKey = (seat: SeatId) => `${seat.reviewer}${seat.shadow ? " shadow" : " /** Whether two reviewers hold the same seat: the same harness, both the shadow or neither. */ export const sameSeat = (a: SeatId, b: SeatId) => seatKey(a) === seatKey(b); -/** The efforts the harnesses take, from lowest to highest. */ -const EFFORTS = ["low", "medium", "high", "xhigh", "max"]; +/** The efforts a reviewer runs at, from lowest to highest; `quest` never runs `max`. */ +const EFFORTS = ["low", "medium", "high", "xhigh"]; /** * The shadow of `routed`, the size's pick: its model one effort lower, or null when no lower effort is known, diff --git a/test/agent.test.ts b/test/agent.test.ts index 30e26637..3fa5ceb0 100644 --- a/test/agent.test.ts +++ b/test/agent.test.ts @@ -300,6 +300,13 @@ test("on Windows, the agent environment drops credentials whatever their case", expect(strippedEnvironment({ op_token: "t" }, "linux")).toEqual({ op_token: "t" }); }); +test("start refuses max effort before launching anything", async () => { + const args = launch("claude").map((arg) => (arg === "high" ? "max" : arg)); + const result = await quest("agent", "start", ...args, ".", "go"); + expect(result.code).toBe(2); + expect(result.stderr).toContain("max is never run"); +}); + test("start runs claude with every launch setting on its own login", async () => { const s = setup(); const result = await quest("agent", "start", ...launch("claude"), s.worktree, "/quest-start quest/m0/a.md"); diff --git a/test/cli.test.ts b/test/cli.test.ts index 1be00ef2..39de53a2 100644 --- a/test/cli.test.ts +++ b/test/cli.test.ts @@ -152,11 +152,11 @@ test("a document that is not UTF-8 is an error", () => { test("unblocked prints ranked counts and paths, with deterministic JSON and a limit", () => { const tree = Tree.baseline(); expect(tree.run("unblocked")).toEqual({ code: 0, stdout: "1\tquest/m0/epic/one.md\n", stderr: "" }); - const opusLow = '{"harness":"claude","model":"claude-opus-5-5","effort":"low"}'; + const opusMedium = '{"harness":"claude","model":"claude-opus-5-5","effort":"medium"}'; const solMedium = '{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}'; expect(tree.run("unblocked", "--json", "--limit", "1")).toEqual({ code: 0, - stdout: `[{"path":"quest/m0/epic/one.md","dependents":1,"hosts":[],"models":[${opusLow},${solMedium}]}]\n`, + stdout: `[{"path":"quest/m0/epic/one.md","dependents":1,"hosts":[],"models":[${opusMedium},${solMedium}]}]\n`, stderr: "", }); tree.write("quest/m0/epic/two.md", "# [S] Two\n\n## Goal\n\nWork.\n"); @@ -175,7 +175,7 @@ test("unblocked routes a quest with a decision open one size up, and XL stays XL quest.models.map((entry) => entry.effort).join(" "), ); const tree = Tree.baseline().append("quest/m0/epic/one.md", "\n## Waiting\n\n- moq-dev/moq: a release.\n"); - expect(models(tree)).toEqual(["low medium"]); + expect(models(tree)).toEqual(["medium medium"]); tree.append("quest/m0/epic/one.md", "- Maintainer: pick a name.\n"); expect(models(tree)).toEqual(["medium high"]); tree.write("quest/m0/epic/one.md", "# [XL] One\n\n## Goal\n\nWork.\n\n## Waiting\n\n- Maintainer: pick.\n"); @@ -195,7 +195,7 @@ test("models ranks a size's writers, or the one entry that runs a PR's writer", const writer = (size: string, model: string) => tree.run("models", size, "--writer", model).stdout; expect(writer("M", "gpt-6.1-sol")).toBe("codex\tgpt-6.1-sol\thigh\n"); // A writer its size does not list runs at the nearest size that lists it. - expect(writer("XS", "claude-opus-5-5")).toBe("claude\tclaude-opus-5-5\tlow\n"); + expect(writer("XS", "claude-opus-5-5")).toBe("claude\tclaude-opus-5-5\tmedium\n"); expect(writer("L", "claude-sonnet-5-5")).toBe("claude\tclaude-sonnet-5-5\thigh\n"); expect(writer("XL", "gpt-6.1-sol")).toBe("codex\tgpt-6.1-sol\txhigh\n"); const unknown = tree.run("models", "S", "--writer", "unknown"); diff --git a/test/review.test.ts b/test/review.test.ts index edd78467..3a074ae5 100644 --- a/test/review.test.ts +++ b/test/review.test.ts @@ -1025,6 +1025,12 @@ fakes("--base reviews against another branch, fetched first, with that branch's expect(unknown.stderr).toContain("--base must be /"); }); +test("review refuses max effort", async () => { + const result = await quest("review", "--harness", "claude", "--model", "opus", "--effort", "max"); + expect(result.code).toBe(2); + expect(result.stderr).toContain("max is never run"); +}); + fakes("claude reviews read-only with the base's instructions, then reports in the schema", async () => { const s = setup("CODEX_THREAD_ID"); // The rules travel inside the command's one double-quoted argument. @@ -1761,7 +1767,7 @@ test("the shadow is the size's pick one effort lower, and none below low", () => "claude claude-opus-5-5 medium", "claude claude-opus-5-5 high", ]); - expect(shadowOf({ harness: "claude", model: "m", effort: "max" })!.effort).toBe("xhigh"); + expect(shadowOf({ harness: "claude", model: "m", effort: "xhigh" })!.effort).toBe("high"); // No step below an effort off the ladder is known, so an --effort such as Codex's minimal runs no shadow. expect(shadowOf({ harness: "codex", model: "m", effort: "minimal" })).toBeNull(); });