Skip to content

test: split the reviewer usage-limit test per case and unskip it - #537

Merged
moq-bot[bot] merged 5 commits into
mainfrom
quest/m0/flaky/a-reviewers-usage-limit-exits-3-with-its-reset-and-records-no-round
Nov 7, 2026
Merged

moq-bot[bot] merged 5 commits into
mainfrom
quest/m0/flaky/a-reviewers-usage-limit-exits-3-with-its-reset-and-records-no-round

Conversation

@kixelated

@kixelated kixelated commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes the flaky test skipped in moq-dev/quest#533.

Root cause: one test ran four cases × two full quest review runs (each spawning fake reviewer CLIs). That took ~2.4s alone and outran the 5000 ms timeout on loaded merge-group runners. Now fakes.for makes one test per case (~0.5–0.6s each), with no timeout, retry, or sleep changes.

Choices made without asking:

  • Deleted the now-childless quest/m0/flaky epic and its link; the queue watcher recreates it when missing (see quest/m0/queue-watcher/flaky-quest.md).

moq-dev/quest#517 edits other parts of test/review.test.ts; expect a trivial merge.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

mechanical fix by codex, gpt-6.1-sol, medium effort: Merged base cleanly and formatted test/review.test.ts without behavior changes; ran just install, bunx --no-install prettier --write test/review.test.ts, just check, just test, and nix flake check --option substituters ''. (head 2759882)

Rewritten by quest review after each round.

  • Reviewer: Codex, gpt-6.1-sol, medium effort
  • Session: 01a1b764-ac59-7043-a598-ca09d6554b69, round 1
  • Head: 60848cb
  • Base: origin/main at 1c3cf6a
  • Score: 10/10, The change preserves the assertions while giving each case its own timeout and the existing Windows skip, and consistently removes the completed quest; tests were not run in the read-only environment.
  • Findings: none

…th-its-reset-and-records-no-round.md

Quest-Claim: 7ac2eaeb9a89df07
@pullfrog

pullfrog Bot commented Nov 7, 2026

Copy link
Copy Markdown

Pullfrog  | Review this ➔

kixelated and others added 2 commits November 7, 2026 10:57
All four cases ran two full reviews each inside one 5s test, which
outran its timeout on loaded merge-group runners. One test per case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Fix the reviewer usage-limit test test: split the reviewer usage-limit test per case and unskip it Nov 7, 2026
@kixelated
kixelated marked this pull request as ready for review November 7, 2026 19:05
@coderabbitai

coderabbitai Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8a170957-cb66-41af-a945-6f4c24f37565
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated kixelated left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review by review (OpenAI)

Reviewed head 60848cb against base 1c3cf6a.

No actionable findings. Splitting the four host/reset combinations into separate fakes.for tests preserves the stderr, exit-code, empty-stdout and subsequent successful-review assertions while giving each pair of review runs its own timeout. The existing fakes wrapper retains the Windows exclusion, each case gets a fresh setup, and afterEach restores stubbed environment variables. Removing the now-empty flaky epic, completed leaf and parent link is consistent with the completed work.

Direction: proceed through normal CI/merge validation; use loaded merge-group runs to confirm the timeout improvement rather than treating the split alone as proof that the flake is eliminated.

Verification limits: GitHub-only static inspection of the diff, test fixtures and flaky-quest handling. I did not run the tests, reproduce runner load, or independently verify the reported timings.

(Written by review (OpenAI))

@kixelated
kixelated added this pull request to the merge queue Nov 7, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 7, 2026
@moq-bot

moq-bot Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

quest land watch: the merge queue removed this pull request at 60848cb because its merge_group build failed (failed_checks).

The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs.

Queue / Queue build / Build (job) failure at step Check:

…oup size are both one in main's ruleset.
�[36;1m# Build concurrency and group size are both one in main's ruleset.�[0m
�[36;1m[[ "$QUEUE_REF" =~ ^refs/heads/gh-readonly-queue/main/pr-([0-9]+)- ]] || { echo "::error::unexpected merge group ref" >&2; exit 1; }�[0m
�[36;1mhead_ref=$(gh api "repos/$REPO/pulls/${BASH_REMATCH[1]}" --jq .head.ref)�[0m
�[36;1m./quest-base/bin/quest --root "$GITHUB_WORKSPACE" check --base "$BASE" --branch "$head_ref"�[0m
shell: /nix/store/1mv3qz005gkbalxfghk4y3s8ayv4d2dl-bash-interactive-5.3p15/bin/bash -e {0}
env:
  BASE: 6b6ce221323fda9350f3dbc84232bd5f5867a3d8
  GH_TOKEN: ***
  REPO: moq-dev/quest
  QUEUE_REF: refs/heads/gh-readonly-queue/main/pr-537-6b6ce221323fda9350f3dbc84232bd5f5867a3d8
##[endgroup]
quest: 77 documents ok
##[group]Run nix develop --command just install
�[36;1mnix develop --command just install�[0m
shell: /nix/store/1mv3qz005gkbalxfghk4y3s8ayv4d2dl-bash-interactive-5.3p15/bin/bash -e {0}
env:
  BASE: 6b6ce221323fda9350f3dbc84232bd5f5867a3d8
##[endgroup]
bun install --frozen-lockfile
bun install v1.4.4 (38afee6c8)

+ @types/mdast@4.0.4
+ @types/node@26.6.6
+ prettier@3.9.10
+ typescript@7.0.2
+ vitest@5.0.5
+ @opencode/client@2.2.1
+ commander@15.0.0
+ mdast-util-from-markdown@2.1.0
+ mdast-util-gfm-strikethrough@2.0.1
+ mdast-util-gfm-table@2.0.0
+ micromark-extension-gfm-strikethrough@2.1.0
+ micromark-extension-gfm-table@2.1.2

187 packages installed [1307.00ms]
##[group]Run nix develop --command just check
�[36;1mnix develop --command just check�[0m
shell: /nix/store/1mv3qz005gkbalxfghk4y3s8ayv4d2dl-bash-interactive-5.3p15/bin/bash -e {0}
env:
  BASE: 6b6ce221323fda9350f3dbc84232bd5f5867a3d8
##[endgroup]
bun run --silent quest gate just _check
bun run check
$ tsc && tsc -p src/core && prettier --check .
Checking formatting...
[�[33mwarn�[39m] test/review.test.ts
[�[33mwarn�[39m] Code style issues found in the above file. Run Prettier with --write to fix.
error: script "check" exited with code 1
error: recipe `_check` failed on line 19 with exit code 1
error: recipe `check` failed on line 16 with exit code 1
##[error]Process completed with exit code 1.
Post job cleanup.
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git version
git version 2.55.0
Temporarily overriding HOME='/var/cache/github-runner-work/_temp/d74ebe12-1a2d-4997-a96d-a9ef13bdf864' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --global --add safe.directory /var/cache/github-runner-work/quest/quest
Removing SSH command configuration
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp core\.sshCommand
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
Removing includeIf entries pointing to credentials config files
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
Post job cleanup.
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git version
git version 2.55.0
Temporarily overriding HOME='/var/cache/github-runner-work/_temp/c42c89f8-95b0-479b-870b-2fa79b9def3b' before making global git config changes
Adding repository directory to the temporary git global config as a safe directory
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --global --add safe.directory /var/cache/github-runner-work/quest/quest
Removing SSH command configuration
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp core\.sshCommand
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
Removing HTTP extra header
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
Removing includeIf entries pointing to credentials config files
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git config --local --name-only --get-regexp ^includeIf\.gitdir:
[command]/nix/store/lg46w7hrjx7kylsh6645c9l975i8h43b-git-2.55.0/bin/git submodule foreach --recursive git config --local --show-origin --name-only --get-regexp remote.origin.url
A job completed hook has been configured by the self-hosted runner administrator
##[group]Run '/nix/store/gx2v6q84b0w43bafm0whkdx93qyxbpps-ci-cache-gc.sh/bin/ci-cache-gc.sh'
shell: /nix/store/1mv3qz005gkbalxfghk4y3s8ayv4d2dl-bash-interactive-5.3p15/bin/bash --noprofile --norc -e -o pipefail {0}
##[endgroup]
evicted 0 objects and 0 action results (0 B logical); 45.9 GiB logical remain
Cleaning up orphan processes

Queue / Queue (job) failure at step Require the build to pass:

Current runner version: '2.338.2'
##[group]Runner Image Provisioner
Hosted Compute Agent
Version: 20261028.624
Commit: 54898b74c23fde5556c9b3c3a2290865feef0dfe
Build Date: 2026-10-28T21:51:42Z
Worker ID: {289274b9-b408-4376-ae0d-16d31d6f8f14}
Region: eastus2
Cloud: Azure
##[endgroup]
##[group]Operating System
Ubuntu
24.04.5
LTS
##[endgroup]
##[group]Runner Image
Image: ubuntu-24.04
Version: 20261101.360.1
Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20261101.360/images/ubuntu/Ubuntu2404-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20261101.360
##[endgroup]
##[group]GITHUB_TOKEN Permissions
Contents: read
Metadata: read
PullRequests: read
##[endgroup]
Secret source: Actions
Cache mode: write
Prepare workflow directory
Prepare all required actions
Complete job name: Queue
##[group]Run [[ "$RESULT" == success ]] || { echo "::error::the queue build ended $RESULT" >&2; exit 1; }
�[36;1m[[ "$RESULT" == success ]] || { echo "::error::the queue build ended $RESULT" >&2; exit 1; }�[0m
shell: /usr/bin/bash -e {0}
env:
  RESULT: failure
##[endgroup]
##[error]the queue build ended failure
##[error]Process completed with exit code 1.
Cleaning up orphan processes

(written by quest land watch)

@moq-bot
moq-bot Bot enabled auto-merge November 7, 2026 20:10
@moq-bot

moq-bot Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

quest land watch: armed auto-merge on this pull request at 2759882, so it joins the back of the merge queue once its required checks pass, with the land fixer's fix (codex, gpt-6.1-sol, medium effort): Merged base cleanly and formatted test/review.test.ts without behavior changes; ran just install, bunx --no-install prettier --write test/review.test.ts, just check, just test, and nix flake check --option substituters ''..

(written by quest land watch)

@moq-bot
moq-bot Bot added this pull request to the merge queue Nov 7, 2026
Merged via the queue into main with commit 455ebea Nov 7, 2026
9 checks passed
@moq-bot
moq-bot Bot deleted the quest/m0/flaky/a-reviewers-usage-limit-exits-3-with-its-reset-and-records-no-round branch November 7, 2026 21:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant