Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ $ quest --root examples/export unblocked
1 quest/m0/export.md

$ quest --root examples/export unblocked --json --limit 1
[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"models":[{"harness":"claude","model":"claude-opus-5-5","effort":"low"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}]
[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"models":[{"harness":"claude","model":"claude-opus-5-5","effort":"medium"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}]
```

Each plain output line contains the count, a tab, and the path. On a terminal,
Expand Down Expand Up @@ -1346,7 +1346,7 @@ the same for every repository unless its `AGENTS.md` names others:
| Size | Writer | Fallback |
|---|---|---|
| XS | Claude Code `claude-sonnet-5-5` high | Codex `gpt-6.1-sol` low |
| S | Claude Code `claude-opus-5-5` low | Codex `gpt-6.1-sol` medium |
| S | Claude Code `claude-opus-5-5` medium | Codex `gpt-6.1-sol` medium |
| M | Claude Code `claude-opus-5-5` medium | Codex `gpt-6.1-sol` high |
| L | Claude Code `claude-opus-5-5` high | Codex `gpt-6.1-sol` xhigh |
| XL | Claude Code `claude-opus-5-5` xhigh | none |
Expand All @@ -1371,14 +1371,14 @@ claude claude-sonnet-5-5 high
codex gpt-6.1-sol low

$ quest models XS --writer claude-opus-5-5
claude claude-opus-5-5 low
claude claude-opus-5-5 medium
```

`--writer` prints the size's entry for that model, or else its entry at the
nearest size that lists it, the larger on a tie; a model no list names is an
error. `--json` prints an array of `{harness, model, effort}` objects. The
iterate records the size it was given in its `Implementer:` line, as
`Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS)`, so
`Implementer: Claude Code, claude-opus-5-5, medium effort (iterate, XS)`, so
the model check can audit the routing. `quest review` still reviews the
whole branch at the PR's size.

Expand All @@ -1399,8 +1399,9 @@ size. Each size ranks its reviewers, the same for every repository:
Beside it, the shadow, the pick's model one effort lower (Sol low for an XS
PR), reviews the same head in its own session, so the routing can be retuned
from what each effort step finds over the one below it. It is skipped when the
pick runs at low or at an `--effort` off the ladder (low, medium, high, xhigh,
max). It runs on the branch's first run of the pick, then reruns only while it has findings to settle, or once the pick
pick runs at low or at an `--effort` off the ladder (low, medium, high, xhigh).

It runs on the branch's first run of the pick, then reruns only while it has findings to settle, or once the pick
changed so that it is no longer one step below; its findings are numbered,
answered, and settled like any reviewer's. A pick with no step below it seats
a shadow at its own effort while the old shadow still has findings, so none
Expand Down Expand Up @@ -1683,6 +1684,10 @@ quest agent stop --all
quest agent admit
```

`--effort` takes any level the CLI does except `max`, which costs about as much
as a stronger model at a lower effort without matching it; `quest review`
refuses it too.

Codex starts and resumes only in a checkout registered by `quest worktree create`,
on Linux with the `quest worktree run` supervisor available.
Ownership and host support are checked before any agent state is written. Each turn uses
Expand Down
28 changes: 24 additions & 4 deletions quest/m0/routing-evidence/writer-ab.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# [M] Split XS, S and M writers between two arms
# [M] Split each size's writers between two arms

## Goal

`quest unblocked --json` routes each XS quest to Sonnet 5.5 high or Opus
low, each S quest to Opus low or medium, and each M quest to Opus medium
or high, 50/50. The split is stable per quest,
`quest unblocked --json` routes each XS and S quest to Sonnet 5.5 high or
Opus 5.5 medium, each M quest to Opus medium or high, and each XL quest to
Opus xhigh or Fable 5.2 medium, 50/50. The split is stable per quest,
and the review efforts in `src/cli/models.ts` stay fixed while it runs. Each `Implementer:` line
records its arm, iterates stay on the PR's arm, and
`quest usage` reports cost, rounds, findings, hand-backs, and later
Expand Down Expand Up @@ -37,6 +37,26 @@ Decided 2026-11-11 (user, /quest-plan,
- Rejected: Sonnet medium against Sonnet high, since the same cost
guide puts Sonnet medium about 7 points below high on its SWE-bench
Pro subset and 15 below on its internal agentic-coding set; no XS arm
- Re-armed 2026-11-13 (user), to test where Anthropic's
[model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/)
and Theo Browne's
[Opus against Fable video](https://www.youtube.com/watch?v=XtykeVoSmP8)
disagree:
- S Sonnet high against Opus medium: the guide puts scoped coding on
Sonnet, Theo finds Sonnet runs up output tokens and costs about what
Opus does. S's table entry moved from Opus low to Opus medium, the
guide's default, since both call Opus low too weak.
- XS Sonnet high against Opus medium rather than Opus low, the same
contrast as S (agent's choice, since no table entry runs Opus low any
more).
- M Opus medium against high, as before: Theo finds medium ends its turn
too early and calls high the best effort.
- XL Opus xhigh against Fable medium: the guide moves up to Fable rather
than past high, while Theo finds Opus checks verifiable work more
carefully and a subscription gives about a quarter as much Fable. XL's
table entry stays Opus xhigh for that subscription cap (rejected:
Fable medium; dropping the Fable arm). XL quests are rare, so report
how many each arm got rather than wait for a detectable difference.
- Escapes are too rare to separate the arms, about 2% of PRs. Compare
first-round findings under the fixed review efforts, rounds, hand-backs,
and cost, and report escapes alongside.
Expand Down
15 changes: 15 additions & 0 deletions quest/m1/model-check.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,21 @@ checkpoints and outages to count by hand:
puts Sonnet 5.5 high level with Opus 5.5 low (87.6% against 87.4%) at
about the same cost per solved task, and medium about 7 points below
high on its SWE-bench Pro subset.
- Changed again (maintainer, 2026-11-13), on Anthropic's
[model guide](https://claude.dev/blog/which-claude-model-should-you-build-with/),
which puts scoped coding on Sonnet 5.5 and advises moving up a model
rather than past `high`: S writes on Opus 5.5 medium instead of low,
whose S PRs averaged 4.7 review rounds over 6 reviewed; and nothing
runs `max`, which `quest agent` and `quest review` refuse. XL stays on
Opus xhigh: the maintainer first moved it to Fable 5.2 medium, then
kept Opus, since a Max or Ultra plan caps Fable at half its weekly
limit and Fable uses that limit faster (Theo Browne's
[video](https://www.youtube.com/watch?v=XtykeVoSmP8) estimates about
$3k a month of Fable against $14k of Opus on the $200 plan). XS stays Sonnet high, since `quest usage` had no
Sonnet medium runs to compare. The
[writer A/B](/quest/m0/routing-evidence/writer-ab.md) tests each change
against its alternative, Fable medium on XL included; tally the S cell
fresh.
- Compare both tables' models (in `src/cli/models.ts`) with each vendor's current models: the newest
model of each tier the tables pin. Bump any that are stale, and check that both CLIs
accept the new ids and the pinned efforts. `--deep` and its strongest-model
Expand Down
19 changes: 16 additions & 3 deletions src/cli/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1048,7 +1048,7 @@ export function main(argv: string[], out: Output): number | Promise<number> {
command
.addOption(new Option("--harness <name>", "Agent CLI to run").choices(HARNESSES).makeOptionMandatory())
.requiredOption("--model <model>", "Model the agent runs")
.requiredOption("--effort <level>", "Reasoning effort the agent runs at")
.requiredOption("--effort <level>", "Reasoning effort the agent runs at, never max", effortLevel)
.requiredOption(
"--permission-mode <mode>",
"Claude Code's --permission-mode, or Codex's sandbox mode; the agent cannot prompt",
Expand Down Expand Up @@ -1231,7 +1231,7 @@ export function main(argv: string[], out: Output): number | Promise<number> {
"Review the checkout's committed branch against its base with a reviewer its size picks, never the " +
`model that changed it last. Each size ranks its reviewers (harness, model, effort): ${reviewers}. The ` +
"first whose model is not avoided reviews, and beside it, on the same head in its own session, the shadow: " +
"the pick's model one effort lower (low, medium, high, xhigh, max), skipped when the pick runs at low or an " +
"the pick's model one effort lower (low, medium, high, xhigh), skipped when the pick runs at low or an " +
"effort off that ladder, so each effort step can be compared with the one below it. The shadow runs on a " +
"branch's first run of the pick, then only while it has findings to settle or once the pick changed so it is " +
"no longer one step below; its findings are numbered and settled like any reviewer's, its session carries " +
Expand Down Expand Up @@ -1317,7 +1317,11 @@ export function main(argv: string[], out: Output): number | Promise<number> {
).choices(HARNESSES),
)
.option("--model <model>", "Model the size's pick, or --harness's reviewer, runs instead")
.option("--effort <level>", "Reasoning effort the size's pick, or --harness's reviewer, runs at instead")
.option(
"--effort <level>",
"Reasoning effort the size's pick, or --harness's reviewer, runs at instead, never max",
effortLevel,
)
.option(
"--exclude-model <model>",
"A model never picked to review it, instead of the PR's latest writer; repeat for each",
Expand Down Expand Up @@ -1637,6 +1641,15 @@ export function main(argv: string[], out: Output): number | Promise<number> {
}
}

/**
* An agent's effort: any level but `max`, which costs as much as a stronger
* model at a lower effort without matching it.
*/
function effortLevel(value: string): string {
if (value === "max") throw new InvalidArgumentError("max is never run; use a stronger model instead");
return nonEmpty(value);
}

/** An empty path names no directory, so it is a usage error rather than `.`. */
function nonEmpty(value: string): string {
if (value === "") throw new InvalidArgumentError("a value is required");
Expand Down
6 changes: 4 additions & 2 deletions src/cli/models.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,13 @@ const sonnet = (effort: string): Model => ({ harness: "claude", model: "claude-s
* writes a PR runs all its iterates. XS runs Sonnet rather than the cheaper
* Haiku, whose surcharge past 100k input tokens leads users to cap its
* compaction window there, barely above an agent's fixed load of prompt,
* context, and skill.
* context, and skill. No size runs Opus at low, which misses too much. XL
* stays on Opus rather than Fable, which a subscription caps at half its
* weekly limit and which uses that limit faster.
*/
export const WRITERS: Record<Size, readonly Model[]> = {
XS: [sonnet("high"), sol("low")],
S: [opus("low"), sol("medium")],
S: [opus("medium"), sol("medium")],
M: [opus("medium"), sol("high")],
L: [opus("high"), sol("xhigh")],
XL: [opus("xhigh")],
Expand Down
4 changes: 2 additions & 2 deletions src/cli/review.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1229,8 +1229,8 @@ const seatKey = (seat: SeatId) => `${seat.reviewer}${seat.shadow ? " shadow" : "
/** Whether two reviewers hold the same seat: the same harness, both the shadow or neither. */
export const sameSeat = (a: SeatId, b: SeatId) => seatKey(a) === seatKey(b);

/** The efforts the harnesses take, from lowest to highest. */
const EFFORTS = ["low", "medium", "high", "xhigh", "max"];
/** The efforts a reviewer runs at, from lowest to highest; `quest` never runs `max`. */
const EFFORTS = ["low", "medium", "high", "xhigh"];

/**
* The shadow of `routed`, the size's pick: its model one effort lower, or null when no lower effort is known,
Expand Down
7 changes: 7 additions & 0 deletions test/agent.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -300,6 +300,13 @@ test("on Windows, the agent environment drops credentials whatever their case",
expect(strippedEnvironment({ op_token: "t" }, "linux")).toEqual({ op_token: "t" });
});

test("start refuses max effort before launching anything", async () => {
const args = launch("claude").map((arg) => (arg === "high" ? "max" : arg));
const result = await quest("agent", "start", ...args, ".", "go");
expect(result.code).toBe(2);
expect(result.stderr).toContain("max is never run");
});

test("start runs claude with every launch setting on its own login", async () => {
const s = setup();
const result = await quest("agent", "start", ...launch("claude"), s.worktree, "/quest-start quest/m0/a.md");
Expand Down
8 changes: 4 additions & 4 deletions test/cli.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -152,11 +152,11 @@ test("a document that is not UTF-8 is an error", () => {
test("unblocked prints ranked counts and paths, with deterministic JSON and a limit", () => {
const tree = Tree.baseline();
expect(tree.run("unblocked")).toEqual({ code: 0, stdout: "1\tquest/m0/epic/one.md\n", stderr: "" });
const opusLow = '{"harness":"claude","model":"claude-opus-5-5","effort":"low"}';
const opusMedium = '{"harness":"claude","model":"claude-opus-5-5","effort":"medium"}';
const solMedium = '{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}';
expect(tree.run("unblocked", "--json", "--limit", "1")).toEqual({
code: 0,
stdout: `[{"path":"quest/m0/epic/one.md","dependents":1,"hosts":[],"models":[${opusLow},${solMedium}]}]\n`,
stdout: `[{"path":"quest/m0/epic/one.md","dependents":1,"hosts":[],"models":[${opusMedium},${solMedium}]}]\n`,
stderr: "",
});
tree.write("quest/m0/epic/two.md", "# [S] Two\n\n## Goal\n\nWork.\n");
Expand All @@ -175,7 +175,7 @@ test("unblocked routes a quest with a decision open one size up, and XL stays XL
quest.models.map((entry) => entry.effort).join(" "),
);
const tree = Tree.baseline().append("quest/m0/epic/one.md", "\n## Waiting\n\n- moq-dev/moq: a release.\n");
expect(models(tree)).toEqual(["low medium"]);
expect(models(tree)).toEqual(["medium medium"]);
tree.append("quest/m0/epic/one.md", "- Maintainer: pick a name.\n");
expect(models(tree)).toEqual(["medium high"]);
tree.write("quest/m0/epic/one.md", "# [XL] One\n\n## Goal\n\nWork.\n\n## Waiting\n\n- Maintainer: pick.\n");
Expand All @@ -195,7 +195,7 @@ test("models ranks a size's writers, or the one entry that runs a PR's writer",
const writer = (size: string, model: string) => tree.run("models", size, "--writer", model).stdout;
expect(writer("M", "gpt-6.1-sol")).toBe("codex\tgpt-6.1-sol\thigh\n");
// A writer its size does not list runs at the nearest size that lists it.
expect(writer("XS", "claude-opus-5-5")).toBe("claude\tclaude-opus-5-5\tlow\n");
expect(writer("XS", "claude-opus-5-5")).toBe("claude\tclaude-opus-5-5\tmedium\n");
expect(writer("L", "claude-sonnet-5-5")).toBe("claude\tclaude-sonnet-5-5\thigh\n");
expect(writer("XL", "gpt-6.1-sol")).toBe("codex\tgpt-6.1-sol\txhigh\n");
const unknown = tree.run("models", "S", "--writer", "unknown");
Expand Down
8 changes: 7 additions & 1 deletion test/review.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1025,6 +1025,12 @@ fakes("--base reviews against another branch, fetched first, with that branch's
expect(unknown.stderr).toContain("--base must be <remote>/<branch>");
});

test("review refuses max effort", async () => {
const result = await quest("review", "--harness", "claude", "--model", "opus", "--effort", "max");
expect(result.code).toBe(2);
expect(result.stderr).toContain("max is never run");
});

fakes("claude reviews read-only with the base's instructions, then reports in the schema", async () => {
const s = setup("CODEX_THREAD_ID");
// The rules travel inside the command's one double-quoted argument.
Expand Down Expand Up @@ -1761,7 +1767,7 @@ test("the shadow is the size's pick one effort lower, and none below low", () =>
"claude claude-opus-5-5 medium",
"claude claude-opus-5-5 high",
]);
expect(shadowOf({ harness: "claude", model: "m", effort: "max" })!.effort).toBe("xhigh");
expect(shadowOf({ harness: "claude", model: "m", effort: "xhigh" })!.effort).toBe("high");
// No step below an effort off the ladder is known, so an --effort such as Codex's minimal runs no shadow.
expect(shadowOf({ harness: "codex", model: "m", effort: "minimal" })).toBeNull();
});
Expand Down
Loading