diff --git a/README.md b/README.md index 4d445923..499d0822 100644 --- a/README.md +++ b/README.md @@ -1356,9 +1356,10 @@ size. Each size ranks its reviewers, the same for every repository: | L | Codex `gpt-6.1-sol` xhigh | Claude Code `claude-opus-5-5` high | | XL | Codex `gpt-6.1-sol` xhigh | Claude Code `claude-opus-5-5` xhigh | -Beside it, opencode `opencode-go/deepseek-v4.1-flash` at `high` reviews the +Beside it, opencode `opencode/deepseek-v4.1-flash-free` at `high` reviews the same head for every size, in its own session, so each head is read by a model -family no writer runs, at about $0.05 a review. The shadow, the pick's model +family no writer runs. It is opencode's free tier of DeepSeek V4.1 Flash, so +reviews spend no paid allowance. The shadow, the pick's model one effort lower (Sol low for an XS PR), reviews it too, in its own session, so the routing can be retuned from what each effort step finds over the one below it. It is skipped when the pick runs at low or at an `--effort` off the ladder @@ -1369,7 +1370,10 @@ answered, and settled like any reviewer's. A pick with no step below it seats a shadow at its own effort while the old shadow still has findings, so none is left unanswered. The reviewers run at once; the first to fail, such as on a usage limit, stops the others, and the round -saves none. +saves none. The one exception is the opencode reviewer's usage limit while it +has no findings of its own to settle: it is left out, as a `skipped` line and +a `Skipped:` line in the record, and the other reviewers' rounds count, so an +outage of the second reviewer never holds a head. A quest branch takes its quest's size, read from the base branch's tip so a change cannot lower its own review; any other PR, such as a skill's own, takes diff --git a/assets/skills/review.md b/assets/skills/review.md index f3188bc7..feb63b89 100644 --- a/assets/skills/review.md +++ b/assets/skills/review.md @@ -41,7 +41,8 @@ Step back before another patch: state the problem the PR solves without referenc Report that comparison, recommending to continue, rethink the approach, merge as is, or drop the PR, and let the user decide. Record each design change, including a rethink the user picks, in a PR comment that starts with `Design change:`, and start the next review with `quest review --new`, which restarts the count. -Either reviewer out of quota or logged out is an outage, not a skip: stop reviewing rather than go on with the other alone, keep the PR a draft, hand it back, and report the outage. +A reviewer out of quota or logged out is an outage, not a skip: stop reviewing rather than go on with the other alone, keep the PR a draft, hand it back, and report the outage. +The exception is the second reviewer's usage limit, which `quest review` reports as a `skipped` line and records itself: the other rounds count, so mention the skip in your summary and go on. Its `## Waiting` item names the reviewer, its error, and the check, such as `Codex: out of quota until 2026-11-03 15:00 UTC; check that quest review runs`, with the reset time `quest review` prints when it exits 3 on a usage limit; a logged-out reviewer waits on `Maintainer:`, with the login command `quest review` prints. Once the reset passes or the login is restored, the item has cleared: whoever finds it runs `/quest-review` on the PR. diff --git a/quest/m1/model-check.md b/quest/m1/model-check.md index c264481c..fa8914fa 100644 --- a/quest/m1/model-check.md +++ b/quest/m1/model-check.md @@ -146,7 +146,7 @@ checkpoints and outages to count by hand: covers all three reviewers: opencode since [opencode reviewers](https://github.com/moq-dev/quest/pull/603), on the model `QUEST_LIVE_OPENCODE_MODEL` names (default - `opencode-go/deepseek-v4.1-flash#high`), with the dev shell's `srt`. Check + `opencode/deepseek-v4.1-flash-free#high`), with the dev shell's `srt`. Check that opencode's `run` still takes no schema (it took none in 2.2.6, so the findings turn's prompt carries it), and that `OPENCODE_DISABLE_PROJECT_CONFIG` still skips the checkout's config and `AGENTS.md`. diff --git a/src/cli/main.ts b/src/cli/main.ts index 1cd95c8b..558cbcb8 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -1217,7 +1217,8 @@ export function main(argv: string[], out: Output): number | Promise { "--effort override that reviewer's, or without --harness, the size pick's; a repository whose AGENTS.md names other reviewers " + "passes them. opencode is no size's pick: --harness opencode needs --model, as provider/model, and " + "--effort, its variant. A --harness run keeps the other reviewers' sessions for the next run of the pair, and refuses while one has findings to settle, as a run of the pair does while a reviewer only --harness runs has some. When the latest writer changes the size's pick to one with no session on the branch, its first session settles the open findings of the reviewer it replaces. The reviewers run at once, and the first to fail stops the others and fails the " + - "run, saving no round. The " + + "run, saving no round, except the opencode reviewer's usage limit while it has no findings of its own to settle: " + + "that leaves it out, printed as a `skipped` line and in the record's `Skipped:` line, and the other rounds count. The " + "size is --size, else the size of the quest a quest/ branch names, read from the base's tip so a " + `change cannot lower its own review, else the PR record's Size line, else ${DEFAULT_SIZE}. ` + "The reviewer reaches its model as the user's own CLI " + @@ -1339,19 +1340,27 @@ export function main(argv: string[], out: Output): number | Promise { }, ); pending = run.then( - async ({ rounds, tallies }) => { + async ({ rounds, tallies, skipped }) => { // The rounds are saved in their sessions either way, so their report prints before a failed record. let record: Recorded | undefined; let failure: unknown; try { - record = await recordRounds(root(), rounds, tallies, github); + record = await recordRounds(root(), rounds, tallies, skipped, github); } catch (error) { failure = error; } if (options.json) - print(JSON.stringify({ rounds, tallies, record: record ?? null, next: nextStep(rounds) })); + print( + JSON.stringify({ + rounds, + tallies, + skipped, + record: record ?? null, + next: nextStep(rounds), + }), + ); else { - for (const line of roundLines(rounds)) print(line); + for (const line of roundLines(rounds, skipped)) print(line); if (record !== undefined) print("url" in record ? `record ${record.url}` : `record skipped: ${record.skipped}`); print(nextStep(rounds)); diff --git a/src/cli/models.ts b/src/cli/models.ts index 0bed0272..0fb60d98 100644 --- a/src/cli/models.ts +++ b/src/cli/models.ts @@ -51,11 +51,12 @@ export const REVIEWERS: Record = { /** * The second reviewer of every PR, beside its size's: DeepSeek through opencode, a model family no writer - * runs, for every size, since a review costs it about $0.05. + * runs, for every size. It runs on opencode's free tier, since on a paid plan a busy day's reviews + * spent its allowance and every review failed until the reset. */ export const SECOND_REVIEWER: Reviewing = { harness: "opencode", - model: "opencode-go/deepseek-v4.1-flash", + model: "opencode/deepseek-v4.1-flash-free", effort: "high", }; diff --git a/src/cli/review-record.ts b/src/cli/review-record.ts index 4c42d521..f31e0ef0 100644 --- a/src/cli/review-record.ts +++ b/src/cli/review-record.ts @@ -5,7 +5,17 @@ import { recorded } from "../github/models"; import { type ClaimDependencies, repositoryPath, upstream } from "./claims"; import { git } from "./git"; import { requireRemote, string } from "./github"; -import { type PullModels, type Round, SEVERITIES, type Tally, located, parseBase, sameSeat } from "./review"; +import { + type PullModels, + type Round, + SEVERITIES, + type Skipped, + type Tally, + located, + parseBase, + resetText, + sameSeat, +} from "./review"; const HARNESSES: Record = { claude: "Claude Code", codex: "Codex", opencode: "opencode" }; @@ -58,9 +68,10 @@ const counted = (tally: Tally): Counts => ({ /** * The `## Review` section's text for a run's `rounds` of the same head, without its heading, with every tally - * on the branch: each round's reviewer's beside its round, and a reviewer the run left out on its own. + * on the branch: each round's reviewer's beside its round, and a reviewer the run left out on its own, with a + * `Skipped:` line when its usage limit left it out. */ -export function reviewRecord(rounds: Round[], tallies: Tally[]): string { +export function reviewRecord(rounds: Round[], tallies: Tally[], skipped: Skipped[] = []): string { const [shared] = rounds; // A reviewer the run left out may still hold open findings, which keep the head from passing too. const clean = @@ -93,6 +104,10 @@ export function reviewRecord(rounds: Round[], tallies: Tally[]): string { ` - ${capitalized(finding.disposition)}: ${finding.severity}, ${code(located(finding))}: ${code(finding.title)}; ${code(finding.reason)}`, ); } + const skip = (seat: Pick) => { + const found = skipped.find((each) => sameSeat(seat, each)); + return found === undefined ? [] : [`- Skipped: usage limit; resets ${resetText(found.until)}`]; + }; for (const tally of tallies.filter((each) => !rounds.some((round) => sameSeat(round, each)))) lines.push( "", @@ -100,6 +115,15 @@ export function reviewRecord(rounds: Round[], tallies: Tally[]): string { // Its session, still named so `quest usage` counts it as reviewing. ...(tally.session === undefined ? [] : [`- Session: ${code(tally.session)}`]), `- Rounds: ${tally.rounds}; ${findingsText(counted(tally))}`, + ...skip(tally), + ); + // A reviewer skipped before its first round has no tally yet. + for (const each of skipped.filter((seat) => !tallies.some((tally) => sameSeat(tally, seat)))) + lines.push( + "", + reviewerLine(each), + `- Rounds: 0; ${findingsText({ findings: { high: 0, medium: 0, low: 0 }, fixed: 0, refuted: 0, settled: 0, open: 0 })}`, + ...skip(each), ); return lines.join("\n"); } @@ -268,12 +292,13 @@ export async function recordRounds( root: string, rounds: Round[], tallies: Tally[], + skipped: Skipped[], deps: ClaimDependencies, ): Promise { const [shared] = rounds; const pull = await openPull(root, shared!.base.ref, shared!.branch, deps); if ("skipped" in pull) return pull; - const next = withReview(pull.body, reviewRecord(rounds, tallies)); + const next = withReview(pull.body, reviewRecord(rounds, tallies, skipped)); if (next !== pull.body) await pull.client.request(pull.path, { method: "PATCH", body: { body: next } }); return { url: pull.url }; } diff --git a/src/cli/review.ts b/src/cli/review.ts index e419f369..3a51ff62 100644 --- a/src/cli/review.ts +++ b/src/cli/review.ts @@ -43,8 +43,7 @@ export class ReviewLimitError extends Error { readonly until: number | null, readonly model: string, ) { - const resets = until === null ? "unknown" : new Date(until).toISOString().replace(/\.\d{3}Z$/, "Z"); - super(`${harness} hit its usage limit; resets ${resets}`); + super(`${harness} hit its usage limit; resets ${resetText(until)}`); } } @@ -324,10 +323,22 @@ export interface Tally { refuted: number; } -/** What a run of `quest review` gives: each reviewer's round, and every tally on the branch, the run's first. */ +/** The second reviewer, left out of a run because its account hit its usage limit, which resets at `until`. */ +export interface Skipped { + reviewer: ReviewHarness; + model: string; + effort: string; + until: number | null; +} + +/** + * What a run of `quest review` gives: each reviewer's round, every tally on the branch, the run's first, and the + * second reviewer when its usage limit left it out. + */ export interface Reviewed { rounds: Round[]; tallies: Tally[]; + skipped: Skipped[]; } /** @@ -1724,9 +1735,21 @@ export async function review(options: ReviewOptions, deps: ReviewDependencies): env: deps.env, signal, }; + // The second reviewer's spent usage limit leaves it out rather than failing the run, so an outage of the + // optional reviewer never blocks a head the size's pick reviewed. It still fails the run while it has findings + // of its own to settle, since only it can settle them. + const skippable = (seat: Seat, error: unknown): error is ReviewLimitError => + error instanceof ReviewLimitError && + seat.reviewer === SECOND_REVIEWER && + !seat.shadow && + (seat.session?.rounds.at(-1)!.findings.length ?? 0) === 0; const results = await Promise.allSettled( seats.map((seat) => - reviewWith(shared, seat, options.responses).catch((error: unknown) => { + reviewWith(shared, seat, options.responses).catch((error: unknown): Skipped => { + if (skippable(seat, error)) { + const { harness: reviewer, model, effort } = seat.reviewer; + return { reviewer, model, effort, until: error.until }; + } controller.abort(error); throw error; }), @@ -1734,13 +1757,24 @@ export async function review(options: ReviewOptions, deps: ReviewDependencies): ); // The first failure stopped the others, so it is the one to report. if (controller.signal.aborted) throw controller.signal.reason; - const done = results.map((result) => (result as PromiseFulfilledResult).value); + const settled = results.map((result) => (result as PromiseFulfilledResult).value); + const ran = seats.flatMap((seat, i) => { + const value = settled[i]!; + return "round" in value ? [{ seat, done: value }] : []; + }); + const skipped = settled.filter((value): value is Skipped => !("round" in value)); + // A skipped reviewer's session, with nothing left to settle, waits for its next run. + const waiting = seats.flatMap((seat, i) => ("round" in settled[i]! || !seat.session ? [] : [seat.session])); const tallies = tallied( options.fresh ? dropped(saved) : (saved?.tallies ?? []), - done.map((each, i) => ({ round: each.round, settles: seats[i]!.session })), + ran.map(({ seat, done }) => ({ round: done.round, settles: seat.session })), ); - save(statePath, { branch, sessions: [...done.map((each) => each.session), ...kept], tallies } satisfies State); - return { rounds: done.map((each) => each.round), tallies }; + save(statePath, { + branch, + sessions: [...ran.map(({ done }) => done.session), ...waiting, ...kept], + tallies, + } satisfies State); + return { rounds: ran.map(({ done }) => done.round), tallies, skipped }; } /** @@ -1995,7 +2029,7 @@ export function nextStep(rounds: Round[]): string { * The plain report: the size, base, and head the reviewers share, then each reviewer's round, one record per * line, and its findings, numbered across the reviewers in order, each with its body indented. */ -export function roundLines(rounds: Round[]): string[] { +export function roundLines(rounds: Round[], skipped: Skipped[] = []): string[] { const [shared] = rounds; const lines = [ `size ${shared!.size} ${shared!.sizeFrom}`, @@ -2033,5 +2067,15 @@ export function roundLines(rounds: Round[]): string[] { ); } } + for (const each of skipped) + lines.push( + "", + `skipped ${each.reviewer} ${each.model} ${each.effort} usage limit; resets ${resetText(each.until)}`, + ); return lines; } + +/** A usage limit's reset as `ReviewLimitError` names it: an ISO 8601 UTC time to the second, or `unknown`. */ +export function resetText(until: number | null): string { + return until === null ? "unknown" : new Date(until).toISOString().replace(/\.\d{3}Z$/, "Z"); +} diff --git a/src/github/models.ts b/src/github/models.ts index 43c83465..621d04b0 100644 --- a/src/github/models.ts +++ b/src/github/models.ts @@ -47,13 +47,14 @@ export function recorded(body: string): { } // Display names and IDs compare alike: `Opus 5.5`, `Claude Opus 5.5`, and `claude-opus-5-5` share a key, as do -// `DeepSeek V4.1 Flash` and `opencode-go/deepseek-v4.1-flash`. +// `DeepSeek V4.1 Flash`, `opencode-go/deepseek-v4.1-flash`, and its free tier's `opencode/deepseek-v4.1-flash-free`. const key = (name: string) => name .toLowerCase() .replace(/^.*\//, "") .replace(/[\s.]+/g, "-") - .replace(/^claude-/, ""); + .replace(/^claude-/, "") + .replace(/-free$/, ""); const KNOWN = new Map( [...Object.values(WRITERS), ...Object.values(REVIEWERS), [SECOND_REVIEWER]] .flat() diff --git a/test/live/review-sandbox.test.ts b/test/live/review-sandbox.test.ts index 44373cac..1fdb4314 100644 --- a/test/live/review-sandbox.test.ts +++ b/test/live/review-sandbox.test.ts @@ -44,7 +44,7 @@ import { openSandbox } from "../../src/cli/sandbox"; * provider/model#variant, or a cheap one OpenCode Go serves. */ const [OPENCODE_MODEL, OPENCODE_EFFORT] = ( - process.env.QUEST_LIVE_OPENCODE_MODEL ?? "opencode-go/deepseek-v4.1-flash#high" + process.env.QUEST_LIVE_OPENCODE_MODEL ?? "opencode/deepseek-v4.1-flash-free#high" ).split("#"); /** The model a sizeless review runs on `harness`. */ diff --git a/test/review.test.ts b/test/review.test.ts index b4ff46ab..38b29f21 100644 --- a/test/review.test.ts +++ b/test/review.test.ts @@ -192,7 +192,7 @@ const instructions = config && fs.readFileSync(config.instructions[0], "utf8"); fs.appendFileSync(process.env.FAKE_LOG + ".opencode", JSON.stringify({ argv, stdin, env, config, instructions, cwd: process.cwd() }) + "\\n"); if (argv[0] === "models") { console.log("opencode/free"); - if (!process.env.FAKE_OPENCODE_LOGGED_OUT) console.log("opencode-go/deepseek-v4.1-flash\\nopencode-go/kimi"); + if (!process.env.FAKE_OPENCODE_LOGGED_OUT) console.log("opencode/deepseek-v4.1-flash-free\\nopencode-go/kimi"); process.exit(0); } const session = argv.includes("--session") ? argv[argv.indexOf("--session") + 1] : "ses_1"; @@ -455,7 +455,7 @@ fakes("a quest branch is reviewed at its quest's size on the base, which its own expect(result.stdout).toMatch(/^size L quest\/m0\/big.md\n/); expect(result.stdout).toContain("\nreviewer codex gpt-6.1-sol xhigh\n"); // The second reviewer runs at its own effort, whatever the size. - expect(result.stdout).toContain("\nreviewer opencode opencode-go/deepseek-v4.1-flash high\n"); + expect(result.stdout).toContain("\nreviewer opencode opencode/deepseek-v4.1-flash-free high\n"); // --size overrides the quest's. const sized = await quest("--root", s.repo, "review", "--size", "S", "--new"); @@ -495,16 +495,16 @@ fakes("--exclude-model picks the reviewer from the PR's implementers instead of "M", "default", ]); - expect(second.model).toBe("opencode-go/deepseek-v4.1-flash"); + expect(second.model).toBe("opencode/deepseek-v4.1-flash-free"); const both = ["--exclude-model", "gpt-6.1-sol", "--exclude-model", "claude-opus-5-5"]; const none = await quest("--root", s.repo, "review", ...both, "--new"); expect(none.code).toBe(2); expect(none.stderr).toContain("every M reviewer (gpt-6.1-sol, claude-opus-5-5) is avoided"); // Nor does the second reviewer review a change it is named as writing. - const deepseek = await quest("--root", s.repo, "review", "--exclude-model", "opencode-go/deepseek-v4.1-flash"); + const deepseek = await quest("--root", s.repo, "review", "--exclude-model", "opencode/deepseek-v4.1-flash-free"); expect(deepseek.code).toBe(2); - expect(deepseek.stderr).toContain("opencode-go/deepseek-v4.1-flash is avoided; pass --harness"); + expect(deepseek.stderr).toContain("opencode/deepseek-v4.1-flash-free is avoided; pass --harness"); }); fakes("an unknown harness is an error, never a same-model review", async () => { @@ -1039,7 +1039,7 @@ fakes("codex reviews read-only with the base's instructions, then reports in the " average([3]) is 0.", "", // The second reviewer reviews the same head, its findings numbered after the first's. - "reviewer opencode opencode-go/deepseek-v4.1-flash high", + "reviewer opencode opencode/deepseek-v4.1-flash-free high", "session ses_1", "round 1", "usage review input 10 cached 0 output 6", @@ -1107,7 +1107,7 @@ fakes("codex reviews read-only with the base's instructions, then reports in the expect(replySchema.properties.earlier).toBeUndefined(); const deepseek = s.opencodeCalls().filter((call) => call.argv[0] === "run"); expect(deepseek).toHaveLength(2); - for (const call of deepseek) expect(call.argv).toContain("opencode-go/deepseek-v4.1-flash#high"); + for (const call of deepseek) expect(call.argv).toContain("opencode/deepseek-v4.1-flash-free#high"); }); fakes("--base reviews against another branch, fetched first, with that branch's instructions", async () => { @@ -1525,7 +1525,7 @@ fakes("each round rewrites the Review section of the branch's open PR, keeping t "- Findings, open until the next round settles them:", " - High, `src/lib.ts:2`: `Off by one`", "", - "- Reviewer: opencode, opencode-go/deepseek-v4.1-flash, high effort", + "- Reviewer: opencode, opencode/deepseek-v4.1-flash-free, high effort", "- Session: `ses_1`, round 1", "- Score: 8/10, `Lands once the off-by-one is fixed.`", "- Rounds: 1; findings 1 (high 1): fixed 0, refuted 0, open 1", @@ -1803,7 +1803,7 @@ test("an Implementer line reads as agents write it", () => { expect(read("Claude Code, Opus 5.5, high effort (iterate, XS, arm B)")).toEqual({ ...opus("high"), size: "XS" }); expect(read("unknown, Opus 5.5, high effort")).toEqual(opus("high", "unknown")); expect(read("Claude Code, Fable 5.2, medium effort")?.model).toBe("claude-fable-5-2"); - expect(read("opencode, DeepSeek V4.1 Flash, high effort")?.model).toBe("opencode-go/deepseek-v4.1-flash"); + expect(read("opencode, DeepSeek V4.1 Flash, high effort")?.model).toBe("opencode/deepseek-v4.1-flash-free"); expect(read("Codex, gpt-6.1-sol, high")?.effort).toBe("high"); expect(read("unknown")).toBeNull(); // A lone field reads as a listed display name or an ID, keeping the size its notes give. @@ -2126,14 +2126,54 @@ fakes("a session saved by an older quest is refused before the reviewer runs", a expect((await quest("--root", s.repo, "review", "--new")).code).toBe(0); }); -fakes("a reviewer's failure stops the other and saves neither round", async () => { +fakes("the second reviewer's usage limit leaves it out, and the size's pick's round counts", async () => { const s = setup(); + vi.stubEnv("FAKE_REPLY", JSON.stringify({ score: 10, rationale: "Lands as is.", findings: [] })); vi.stubEnv("FAKE_OPENCODE_LIMITED", "reset"); const result = await quest("--root", s.repo, "review"); + expect(result.code, result.stderr).toBe(0); + expect(result.stdout).not.toContain("\nreviewer opencode "); + expect(result.stdout).toContain( + "\nskipped opencode opencode/deepseek-v4.1-flash-free high usage limit; resets 2030-03-17T17:46:40Z\n", + ); + expect(result.stdout).toContain("next done: no findings remain on this head"); + vi.stubEnv("FAKE_OPENCODE_LIMITED", undefined); + // Once its limit resets, it reviews the head beside the pick again. + const back = await quest("--root", s.repo, "review"); + expect(back.code, back.stderr).toBe(0); + expect(back.stdout).toContain("\nreviewer opencode opencode/deepseek-v4.1-flash-free high\n"); +}); + +fakes("the second reviewer's usage limit still fails the run while it has findings to settle", async () => { + const s = setup(); + const found = await quest("--root", s.repo, "review"); + expect(found.code, found.stderr).toBe(0); + const state = JSON.parse(readFileSync(sessionFile(join(s.repo, ".git"), "feature"), "utf8")); + const own = state.sessions.find((each: { reviewer: string }) => each.reviewer === "opencode"); + expect(own.rounds.at(-1).findings.length).toBeGreaterThan(0); + const numbers = state.sessions.flatMap( + (each: { rounds: { findings: unknown[] }[] }) => each.rounds.at(-1)!.findings, + ); + const responses = join(s.repo, "..", "responses.json"); + writeFileSync( + responses, + JSON.stringify( + numbers.map((_: unknown, i: number) => ({ finding: i + 1, disposition: "fixed", reason: "Fixed." })), + ), + ); + vi.stubEnv("FAKE_OPENCODE_LIMITED", "reset"); + const result = await quest("--root", s.repo, "review", "--responses", responses); expect(result.stderr).toBe("error: opencode hit its usage limit; resets 2030-03-17T17:46:40Z\n"); expect(result.code).toBe(3); +}); + +fakes("a reviewer's failure stops the other and saves neither round", async () => { + const s = setup(); + vi.stubEnv("FAKE_OPENCODE_REJECTED", "1"); + const result = await quest("--root", s.repo, "review"); + expect(result.code).not.toBe(0); expect(existsSync(sessionFile(join(s.repo, ".git"), "feature"))).toBe(false); - vi.stubEnv("FAKE_OPENCODE_LIMITED", undefined); + vi.stubEnv("FAKE_OPENCODE_REJECTED", undefined); // No round was saved, so the next one needs no responses. expect((await quest("--root", s.repo, "review")).code).toBe(0); }); diff --git a/test/usage.test.ts b/test/usage.test.ts index 75335418..555e88c4 100644 --- a/test/usage.test.ts +++ b/test/usage.test.ts @@ -309,6 +309,36 @@ describe("PR records", () => { expect(fixes(body)).toEqual(["codex, gpt-6.1-sol, medium effort"]); }); + test("a reviewer its usage limit left out is recorded with its reset, with or without a tally", () => { + const clean = { ...second, reviewer: "codex" as const, model: "gpt-6.1-sol" }; + const until = Date.UTC(2030, 2, 17, 17, 46, 40); + const skipped = [ + { reviewer: "opencode" as const, model: "opencode/deepseek-v4.1-flash-free", effort: "high", until }, + ]; + const first = reviewRecord([clean], [tally("codex", 1, { high: 0, medium: 0, low: 0 }, 0, 0)], skipped); + expect(first).toContain( + "\n- Reviewer: opencode, opencode/deepseek-v4.1-flash-free, high effort\n- Rounds: 0; findings 0: fixed 0, refuted 0, open 0\n- Skipped: usage limit; resets 2030-03-17T17:46:40Z", + ); + // It reports no finding, so the head the pick passed is named in full. + expect(first).toContain(`- Head: ${clean.head}\n`); + const later = reviewRecord( + [clean], + [ + tally("codex", 2, { high: 0, medium: 0, low: 0 }, 0, 0), + { ...tally("opencode", 1, { high: 0, medium: 0, low: 0 }, 0, 0), session: "ses_1" }, + ], + [{ ...skipped[0]!, until: null }], + ); + expect(later).toContain( + "- Session: `ses_1`\n- Rounds: 1; findings 0: fixed 0, refuted 0, open 0\n- Skipped: usage limit; resets unknown", + ); + for (const record of [first, later]) + expect(parseReview(`## Review\n\n${record}\n`).map((each) => [each.harness, each.round])).toEqual([ + ["Codex", 1], + ["opencode", null], + ]); + }); + test("a record written before tallies has none, and a Rounds line that does not add up fails", () => { const record = reviewRecord([second], [tally("opencode", 1, { high: 0, medium: 0, low: 0 }, 0, 0)]); const older = `## Review\n\n${record.replace(/^- Rounds: .*\n/m, "")}\n`;