Skip to content

feat(models)!: S writes on Opus medium, A/B the disputed writer picks, and refuse max effort - #702

Merged
kixelated merged 5 commits into
mainfrom
routing/no-max-sonnet-s-fable-xl
Nov 13, 2026
Merged

kixelated merged 5 commits into
mainfrom
routing/no-max-sonnet-s-fable-xl

Conversation

@kixelated

@kixelated kixelated commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Retunes the writer table from Anthropic's model guide and Theo Browne's Opus-against-Fable video, refuses max effort everywhere quest takes one, and sets up A/B tests of the disputed picks.

Changes

  • S writes on Opus 5.5 medium instead of Opus low. Both sources call low too weak, and quest usage since 2026-10-01 has Opus low S PRs averaging 4.7 review rounds over 6 reviewed.
  • XL stays on Opus xhigh. The guide advises moving up to Fable past high, but a Max or Ultra plan caps Fable at half its weekly limit, and Fable uses that limit faster. The writer A/B measures Fable medium on XL instead.
  • max is refused: quest agent start|resume --effort max and quest review --effort max fail as usage errors (exit 2), and the review shadow ladder stops at xhigh. quest land fix --effort never accepted it.
  • XS stays on Sonnet high: quest usage has no Sonnet medium runs, and Anthropic's cost guide puts medium about 7 points below high.
  • Writer A/B is re-armed to test where the two sources disagree: XS and S Sonnet high against Opus medium, M Opus medium against high, and XL Opus xhigh against Fable medium.
  • Model check records the change for the next tally.

Agent's choice for review: the XS arm moves from Opus low to Opus medium, since no table entry runs Opus low any more.

Public API

  • Breaking: --effort max is rejected by quest agent and quest review. To migrate, pass xhigh, or a stronger model at a lower effort.
  • quest models and quest unblocked --json print the new S entry; quest models <size> --writer claude-opus-5-5 now resolves XS to Opus medium.

Validation

  • just check and just test pass (1318 tests).
  • Ran the built CLI from this checkout: quest models S prints Opus medium then Sol medium; quest models XL prints Opus xhigh; the README's unblocked --json and models XS --writer examples match the output. quest agent start ... --effort max and quest review ... --effort max exit 2 with max is never run; use a stronger model instead.
  • quest review ran Codex alone (--harness codex): the DeepSeek second reviewer was out of quota, and the maintainer chose Pullfrog's online review in its place, as moq-dev/quest#701 makes standard.

Decisions

  • Follow-up "Hand looping PRs to Fable in half the cases": ✅ drop (rejected: keep). Its quest file is deleted, so its threshold question is moot.

Models

  • Size: S
  • Implementer: Claude Code, claude-opus-5-5, unknown effort
  • Implementer: Claude Code, claude-opus-5-5, low effort (iterate, S)

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: ea50b47

  • Base: origin/main at cd70476

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1d6a7-842f-7b10-8baf-df275bcb46a5, round 3

  • Score: 9/10, The only new branch-specific update clarifies exact-model reviewer avoidance consistently with the code; no actionable defects were found, and builds and tests were not run as instructed.

  • Rounds: 3; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, low effort, shadow

  • Session: 01a1d6ce-9e47-7dc3-a633-a22874c8413d, round 1

  • Score: 9/10, The routing update and max-effort rejection match the documentation and tests, with no actionable defects found by inspection; builds and tests were not run as instructed.

  • Rounds: 1; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

Review skipped for fe98400: it only deletes the dropped follow-up quest, quest/m0/routing-evidence/fable-unstick.md, after the reviewed head ea50b47.

kixelated and others added 2 commits November 13, 2026 12:38
…ng runs max

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 13, 2026 20:44
@coderabbitai

coderabbitai Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The CLI now rejects max for agent and review effort options. The shadow effort ladder ends at xhigh, and the S-size Opus writer uses medium effort. README examples and tests reflect these changes. Routing documents describe writer A/B comparisons and a proposed Fable experiment.

Merge Risk: Minimal · up to dd600

This change rejects --effort max with a usage error, raises the S-size writer to medium effort, and updates the routing docs. I found no concrete defects, and the Fable experiment exists only as a plan, so the merge risk is minimal.

Pre-merge checks: Passed 4

Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title names the main change: S writes move to Opus medium, the disputed writer picks get A/B tests, and max effort is refused. It is a long, multi-part title but it is related to the changeset and…
Description check Passed The description is specific and tied to the changeset. It says the writer routing table is retuned for S and XL, max effort is refused by quest agent and quest review, and the writer A/B and mod…
Full details: Title check

Explanation

The title names the main change: S writes move to Opus medium, the disputed writer picks get A/B tests, and max effort is refused. It is a long, multi-part title but it is related to the changeset and specific enough to scan history.

Full details: Description check

Explanation

The description is specific and tied to the changeset. It says the writer routing table is retuned for S and XL, max effort is refused by quest agent and quest review, and the writer A/B and model-check docs are updated. It also lists the validation steps run and the public API impact, such as the exit 2 usage errors and the new quest models output for S and XS.

Finishing Touches

Simplify code

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c518a9e2-4404-448c-8e8a-ea163dc99a03
Commits

Reviewing files that changed from the base of the PR and between 84f430c and dd6007d.

Files selected for processing: 10

  • README.md
  • quest/m0/routing-evidence/fable-unstick.md
  • quest/m0/routing-evidence/writer-ab.md
  • quest/m1/model-check.md
  • src/cli/main.ts
  • src/cli/models.ts
  • src/cli/review.ts
  • test/agent.test.ts
  • test/cli.test.ts
  • test/review.test.ts

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one minor wording point inline.

Reviewed changes

Reviewed the full diff: the writer table retune, the max effort refusal, and the accompanying quest and README updates.

  • S writer: S now writes on claude-opus-5-5 medium. The README, unblocked --json, models XS --writer examples and tests all agree.
  • XL writer: XL writes on claude-fable-5-2 medium through a new fable helper, with Opus xhigh as its fallback.
  • max refused: the new effortLevel parser rejects max on quest agent start|resume and quest review with exit 2. The shadow EFFORTS ladder stops at xhigh. Both paths have tests.
  • Quest docs: writer-ab.md is re-armed, model-check.md records the change, and the new fable-unstick.md follow-up is added.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title feat(models)!: retune writers from Anthropic's model guide, A/B the disputed picks, and refuse max effort feat(models)!: S writes on Opus medium, A/B the disputed writer picks, and refuse max effort Nov 13, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the delta since the prior pullfrog review (806a0f6): the XL writer reverts to Opus xhigh.

  • XL writer table: WRITERS.XL is [opus("xhigh")] again and the now-unused fable helper is gone, with the doc comment explaining the subscription cap.
  • Docs and tests: README table, writer-ab.md (Fable medium is only the A/B arm), model-check.md, and test/cli.test.ts expectations all agree with the table.

The earlier minor wording point on fable-unstick.md is unchanged and still open.

Pullfrog  | Fix it ➔ | View workflow run | Using claude-sonnet-5-5 | 𝕏

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 13, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

Reviewed the delta since the prior pullfrog review (dd6007d): a merge of main and a wording fix.

  • Merged main: Folded in #701's removal of the second reviewer. The quest review help text and the README shadow paragraph drop SECOND_REVIEWER and max, and --harness now uses HARNESSES.
  • Corrected fable-unstick.md: It now says reviewer avoidance matches the exact model id, which settles the earlier wording point.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes since the prior Pullfrog review:

  • Dropped the fable-unstick follow-up quest. Deleted quest/m0/routing-evidence/fable-unstick.md. A repo-wide grep finds no remaining references to it.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 13, 2026
Merged via the queue into main with commit c1ff799 Nov 13, 2026
19 checks passed
@kixelated
kixelated deleted the routing/no-max-sonnet-s-fable-xl branch November 13, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant