Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 23 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -132,14 +132,15 @@ $ quest --root examples/export unblocked
1 quest/m0/export.md

$ quest --root examples/export unblocked --json --limit 1
[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"models":[{"harness":"claude","model":"claude-opus-5-5","effort":"medium"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}]
[{"path":"quest/m0/export.md","dependents":1,"hosts":[],"arm":"B","models":[{"harness":"claude","model":"claude-sonnet-5-5","effort":"high"},{"harness":"claude","model":"claude-opus-5-5","effort":"medium"},{"harness":"codex","model":"gpt-6.1-sol","effort":"medium"}]}]
```

Each plain output line contains the count, a tab, and the path. On a terminal,
a heading explains the count. `--json` prints an ordered array, including `[]`
when no quests are ready, and gives each quest the `hosts` its `## Hosts`
section names (empty when it runs anywhere) and the ranked `models` that write
it, as [Route work by size](#route-work-by-size) describes; `--limit <count>`
section names (empty when it runs anywhere), its writer A/B `arm`, and the
ranked `models` that write it, as [Route work by size](#route-work-by-size)
describes; `--limit <count>`
requires a positive integer.
Invalid trees fail with diagnostics and no ranking.
`quest-run` uses the ranking unless you explicitly select work in another order.
Expand Down Expand Up @@ -1273,6 +1274,19 @@ to the next only while nothing is written: when the CLI is missing or logged
out, or the agent stops at its usage limit with nothing beyond the claim
commit. Once a model writes, the PR is its own.

While the writer A/B runs, XS, S, M, and XL quests split 50/50 between two
arms, to measure where published guidance disagrees: arm A runs the table's
writer, and arm B runs Claude Code `claude-opus-5-5` medium for XS,
`claude-sonnet-5-5` high for S, `claude-opus-5-5` high for M, and
`claude-fable-5-2` medium for XL. A hash of the quest's path picks the arm,
so a restart or respawn keeps it; `quest unblocked --json` names it as `arm`,
`A` or `B`, and puts its entry first in `models`. L runs no arm, nor does a
quest with a `Maintainer:` decision open, which runs the next size's list, so
an arm always runs at its PR's size; their `arm` is `null`. The agent
that runs that entry ends its `Implementer:` line with the arm, as
`Implementer: Claude Code, claude-opus-5-5, high effort (arm B)`; a fallback
names none, so its PR counts in neither arm.

A `/quest-iterate` stays on the PR's writer, the model its `## Models`
record's first `Implementer:` line names, so the other model can always
review it. The coordinator dispatching it sizes the work left on the PR, not
Expand All @@ -1290,7 +1304,9 @@ claude claude-opus-5-5 medium

`--writer` prints the size's entry for that model, or else its entry at the
nearest size that lists it, the larger on a tie; a model no list names is an
error. `--json` prints an array of `{harness, model, effort}` objects. The
error. An iterate on a PR whose first `Implementer:` line names an arm runs
that arm's entry at the PR's size instead, as `quest models M --arm B`
prints, and records it as `(iterate, XS, arm B)`. `--json` prints an array of `{harness, model, effort}` objects. The
iterate records the size it was given in its `Implementer:` line, as
`Implementer: Claude Code, claude-opus-5-5, medium effort (iterate, XS)`, so
the model check can audit the routing. `quest review` still reviews the
Expand Down Expand Up @@ -1612,7 +1628,9 @@ findings measure an effort step rather than the writer. Since a branch's tallies
PR whose record holds its latest tally, by the most rounds and then the newest,
counts them, and a record written before tallies
counts as unreviewed. An `Implementer:` line's own size, as
`(iterate, XS)`, counts over the PR's; each risk tier's PRs, how many merged,
`(iterate, XS)`, counts over the PR's; the writer A/B by size and arm, with
the same outcomes and every session's spend, counting a sized PR only when
each of its `Implementer:` lines names the same arm; each risk tier's PRs, how many merged,
how many a reviewer raised, and how many a later PR's `## Caused by` names,
which shows whether the tiers predict what breaks; and, over the merged PRs, each online
reviewer's coverage, final heads reviewed, findings per PR reviewed, and
Expand Down
2 changes: 1 addition & 1 deletion assets/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,7 +215,7 @@ Leave the worktree you are running in and any holding uncommitted work or refuse

Run each agent at an entry of `quest`'s ranked lists unless the repository's `AGENTS.md` names other models: as a native sub-agent on this session's harness, else through `quest agent`.
Size each `/quest-iterate` you dispatch as a quest, by the work left on its PR, and name that size in its brief.
Start it on the PR's writer, the model of its first `Implementer:` line, at the entry `quest models <size> --writer <model>` prints, waiting out its usage limit so the other model can review; a PR whose writer no list names, such as `unknown` or a retired model, takes its size's first entry that can run.
Start it on the PR's writer, the model of its first `Implementer:` line, at the entry `quest models <size> --writer <model>` prints, or `quest models <PR size> --arm <arm>` when that line names a writer A/B arm, as `(arm B)`, naming the arm in its brief, waiting out its usage limit so the other model can review; a PR whose writer no list names, such as `unknown` or a retired model, takes its size's first entry that can run.

`quest agent start --harness <claude|codex> --model <model> --effort <level> --permission-mode <mode> <worktree> <prompt>` runs a headless agent on that CLI's own login; start it as a background command in its own isolated worktree, so the harness reports its exit like a sub-agent's.
`--permission-mode` is Claude Code's permission mode, or Codex's sandbox mode.
Expand Down
2 changes: 1 addition & 1 deletion assets/skills/iterate.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Take it with `quest iterate <pr> [token]` before acting, passing a hold token yo
If it refuses, stop and report what it printed; taking over a hold is the user's choice.
If you cannot push to the PR's branch, stop and report.
When another worktree holds the PR's branch, work in a detached checkout of its head.
Your `Implementer:` line ends with the size your brief gives the work, as `Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS)`.
Your `Implementer:` line ends with the size your brief gives the work, and the arm it names, as `Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS, arm B)`.

Read the PR, its linked issues and quest, the reviews and comments posted on it, and the surrounding code.
`quest status <pr>` reports its checks, reviews by head, and unresolved threads in one call.
Expand Down
1 change: 1 addition & 0 deletions assets/skills/run.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Skip a quest a match already implements; one that only touches the same files is
Each agent runs `/quest-start`; its `quest start` refusing a quest claimed since is a skip, not a failure.

Start each on the first entry of its `models` that can run.
When it has an `arm` and runs that first entry, its brief says to end its `Implementer:` line with the arm, as `(arm B)`.
Fall to the next when `quest agent` reports its CLI missing or logged out, or the agent stops on a usage limit with nothing written beyond the claim commit, continuing in that checkout and claim PR.
Once a model has written anything, the PR is its: keep its worktree and session id to resume after the reset.
Skip a quest whose list runs out, and report each harness's outage once.
Expand Down
14 changes: 11 additions & 3 deletions quest/m0/routing-evidence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,13 @@ Readout, the agent's choices for the implementer to revisit:
- Start one week after both experiments are live. From `quest usage --json`,
compute each arm's variance and the number of PRs each arm needs to
detect a difference worth acting on. If the arms have not reached it,
rewrite the `Waiting` item below for the date they will, and stop.
rewrite the `Waiting` item below for the date they will, and stop. XL
quests are rare, so report how many each XL arm got rather than wait for
a detectable difference there (user, 2026-11-13).
- `quest usage`'s `arms` rollup counts each sized PR whose every
`Implementer:` line names the same arm, under its first line's writer,
with every session's spend on it. Escapes are too rare to separate the
arms (about 2% of PRs): report them alongside, not as the measure.
- For each writer arm, compare first-round findings from the fixed efforts in
`src/cli/models.ts`, review rounds, hand-backs, later `## Caused by` culprits, and cost
per merged PR. For each shadow pair, compare findings the routed reviewer
Expand All @@ -54,7 +60,9 @@ Readout, the agent's choices for the implementer to revisit:
each finding's location from the branch's saved rounds under the clones'
`.git/quest/review/`. Recommend each table change to the
maintainer with its tally. End the writer A/B by deleting its arms from
`src/cli/models.ts`. Keep the shadow reviewer only if the maintainer
`src/cli/models.ts`, with `quest models --arm`, the `arm` field of
`quest unblocked --json`, and the arm notes in the README, guide, and
`/quest-run` and `/quest-iterate`. Keep the shadow reviewer only if the maintainer
chooses to.
- Carry the tallies into [the model check](/quest/m1/model-check.md)'s
baseline.
Expand Down Expand Up @@ -106,7 +114,7 @@ Baseline from the 2026-11-11 audit, at API list price:
- moq-dev/quest: one week of PRs since both the shadow reviewer
(`quest review`'s, PR titled "quest: run a shadow reviewer one effort
below the routed one") and
[the writer A/B](/quest/m0/routing-evidence/writer-ab.md) merged; check
the writer A/B ([moq-dev/quest#720](https://github.com/moq-dev/quest/pull/720)) merged; check
their merge dates with `gh pr list --state merged` and the day with `date`

## Related
Expand Down
2 changes: 1 addition & 1 deletion quest/m0/routing-evidence/charts.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ Decided 2026-11-11 (user, /quest-plan in a /quest-run session, when picking

The benchmarks behind the XS writer change to Sonnet 5.5 high
([moq-dev/quest#646](https://github.com/moq-dev/quest/pull/646)) and the XS
arm of the [writer A/B](/quest/m0/routing-evidence/writer-ab.md)
arm of the writer A/B ([moq-dev/quest#720](https://github.com/moq-dev/quest/pull/720))
([moq-dev/quest#644](https://github.com/moq-dev/quest/pull/644)), both
decided on 2026-11-11, for the first table:

Expand Down
77 changes: 0 additions & 77 deletions quest/m0/routing-evidence/writer-ab.md

This file was deleted.

2 changes: 1 addition & 1 deletion quest/m1/model-check.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ checkpoints and outages to count by hand:
[video](https://www.youtube.com/watch?v=XtykeVoSmP8) estimates about
$3k a month of Fable against $14k of Opus on the $200 plan). XS stays Sonnet high, since `quest usage` had no
Sonnet medium runs to compare. The
[writer A/B](/quest/m0/routing-evidence/writer-ab.md) tests each change
writer A/B ([moq-dev/quest#720](https://github.com/moq-dev/quest/pull/720)) tests each change
against its alternative, Fable medium on XL included; tally the S cell
fresh.
- Compare both tables' models (in `src/cli/models.ts`) with each vendor's current models: the newest
Expand Down
Loading
Loading