Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@ Do not edit `CONTRIBUTING.md`, `assets/CONTEXT.md`, `assets/AGENTS.md`, or
PR, or the standing one /quest-run gives a PR with no user-facing behavior
change that adds or expands no quests; and only after repository checks and
reviews pass.
- A change to the reviewers in `src/cli/models.ts`, the review prompts in
`src/cli/review.ts`, or a `REVIEW.md` runs `quest eval run` with the base's
build and the head's, and puts both scores and their cost in its PR's
`## Validation`.
- Pull requests land as squash merges.
- Do not comment on repositories outside kixelated/moq-dev without approval.

Expand Down
27 changes: 27 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1558,6 +1558,33 @@ the reason of the first to judge it so. The record names it after the base:
tier, and a reviewer that raised it adds `- Raised the risk to high: <reason>`
to its block.

### Score the reviewer

`quest eval` scores the reviewer on the repository's past PRs, kept in
`review-eval.json` at its root: culprits, each with the places at its last head
of the defect a later fix's `## Caused by` named, and clean PRs that broke
nothing. Score a change to the reviewer table, the prompts, or `REVIEW.md`
before it lands, and put the score and its cost in its PR.

```sh
quest eval add 570 --fix 704 --what "It counts every link as a culprit." \
--defect src/cli/usage.ts:395-399 --defect src/cli/usage.ts:408-410
quest eval add 503 --clean
quest eval run # the size's pick and its shadow on every case
quest eval run --harness claude --effort high --pr 570
```

`add` reads the PR's last head and base from GitHub and records its merge base,
size, latest writer, and tier, and a culprit already in the set gains each
further defect. `run` fetches each case from the set's repository, so a fork's
clone scores its upstream's set, and reviews it at its head in a detached checkout,
giving the reviewers the first round `quest review` gives, with the base's
`AGENTS.md` and `CLAUDE.md` and this checkout's committed `REVIEW.md`. A
finding on any line of a defect's places finds it; every finding on a clean PR
is noise. It prints defects found and clean findings by severity for the pick,
its shadow, and either, by size and by tier, each case's rounds, and the cost
token-max prices for the run's sessions.

## Audit cost and accuracy

`quest usage` reports `origin`'s PRs updated since `--since` (a local
Expand Down
40 changes: 0 additions & 40 deletions quest/m0/benchmark/reviewer-eval.md

This file was deleted.

5 changes: 5 additions & 0 deletions quest/m0/mistake-audit.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,11 @@ Agent's choices, for the implementer to revisit. Each run:
owns that gate, so it goes in the run PR's `## Follow-ups` for
`/quest-plan` there, as the guide's Creation section says (user,
2026-11-11, [routing evidence](/quest/m0/routing-evidence/README.md)).
- Add each escape classified catchable or maybe to its repository's eval
set with `quest eval add`, its defect located at the culprit's head from
the fix's diff and confirmed against the culprit's own diff, so `quest
eval run` scores reviewers on it. A private repository's set stays in
that repository.
- Before proposing a fix, find the quest, check, or rule that already owns
the class, and report whether its count fell since that landed. A rule
whose class still recurs is the case for moving it up a level.
Expand Down
5 changes: 3 additions & 2 deletions quest/m1/model-check.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,9 @@ checkpoints and outages to count by hand:
- Move a size to a stronger model or effort where its PRs needed clearly
more rounds or fixes than the size above, and to a cheaper one where its
reviews come back clean. Do the same for the review efforts in `src/cli/models.ts`
from what later reviews and follow-ups found that a review missed. Record the tallies in
the next run's Plan as its baseline.
from what later reviews and follow-ups found that a review missed, and score each
reviewer change with `quest eval run` before recommending it (`README.md`'s Score the
reviewer). Record the tallies in the next run's Plan as its baseline.
- Recommend each change to a list's order, harness, or effort to the
maintainer with its tally, as a decision in the PR, rather than changing
it unasked. Replacing a stale model id with the same vendor's newer one,
Expand Down
Loading
Loading