Skip to content

quest: plan the verification rule and a weekly mistake audit - #605

Draft
kixelated wants to merge 1 commit into
mainfrom
quest-plan/rework-audit
Draft

kixelated wants to merge 1 commit into
mainfrom
quest-plan/rework-audit

Conversation

@kixelated

@kixelated kixelated commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Plans three m0 quests from a comparison of Quest with pstack and a scan of the maintainer's typed messages (Claude Code and Codex, 2026-10-26 to 2026-11-10).

  • quest/m0/real-surface.md (S): one prose rule in quest context that passing checks is not verification. Only moq-dev/moq.pro carries it today.
  • quest/m0/typed-messages.md (M): a quest command that prints only the messages a person typed, so an audit stops rebuilding the extraction.
  • quest/m0/mistake-audit.md (L): a weekly recurring audit that groups repeated mistakes into classes and adds a follow-up quest per class. It requires the command, and its Plan holds the scan as its baseline.

Evidence: since 2026-11-06, 65 of 416 merged moq-dev/moq PRs and 14 of 80 merged moq-dev/quest PRs carry "Caused by" in the body (a phrase search, so approximate). The scan was the third hand-run transcript audit in six days.

Decisions

Chosen options marked ✅; each is also in its quest's Plan.

  • Goal: ✅ the verification rule and the recurring audit (rejected: either alone).
  • A rule that agents settle by experiment what running something would answer: ✅ dropped, since no typed message supported it; the audit's first run tallies how often an answer matched the recommendation (rejected: plan it as stated; ask less by default).
  • Audit inputs: ✅ forge records and the maintainer's typed messages through a quest command (rejected: forge records only; a script each run).
  • Enforcing verification: ✅ prose only (rejected: a checked ## Validation record that quest ready refuses without, which the agent recommended; a reviewer finding only).
  • Audit cadence: ✅ weekly, as a ## Waiting item (rejected: each milestone; each release cut).
  • Milestone: ✅ all three in m0 (rejected: verification in m0 and the audit in m1; all in m1).

Agent's choices, for review:

  • Three flat quests rather than an epic: the audit re-measures itself each run, so no epic README has work of its own.
  • The audit publishes counts and classes in its own words, never a message's text, since this repository is public.
  • The audit edits no AGENTS.md, CONTRIBUTING.md, or skill; each class becomes a follow-up quest for the maintainer's pick.
  • The scan's counts in the baseline came from three sub-agents and were not rechecked.

Follow-ups

None.

Impact

Public API: none. Wire: none. Only quest/ changes.

Validation

quest check passes on the tree (36 documents). quest gate just check did not run: see Waiting. just test did not run: no code changed.

Waiting

  • Host: quest gate just check gets room. Its last attempt waited two hours without starting, behind another session's gated mutation campaign (1-minute load 36 on 24 CPUs at the last look). An earlier attempt did start and failed only because this checkout had no dependencies installed, which just install fixed. Check by running it in this branch, then run /quest-review.

Models

Size: M
Implementer: Claude Code, claude-fable-5-2, high effort

🤖 Generated with Claude Code

(Written by Claude Fable 5 with high)

Three m0 quests from a comparison with pstack and a scan of the maintainer's typed messages: a prose rule that passing checks is not verification, a command that prints typed messages, and a recurring audit that turns repeated mistakes into follow-up quests.

Co-Authored-By: Claude <noreply@anthropic.com>
@pullfrog

pullfrog Bot commented Nov 10, 2026

Copy link
Copy Markdown

Pullfrog  | Review this ➔

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant