Skip to content

fix(gate): signal a scope from the gate, not a helper the enclosing scope kills - #629

Merged
kixelated merged 4 commits into
mainfrom
quest/m0/gate-test-races
Nov 11, 2026
Merged

kixelated merged 4 commits into
mainfrom
quest/m0/gate-test-races

Conversation

@kixelated

@kixelated kixelated commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes Gate tests pass deterministically, deleting it and its links.

Summary

Both intermittent failures in test/gate.test.ts are reproduced and fixed at their cause:

  • concurrent processes never admit past the headroom, and admit every caller: the exit-code mismatch in run 42676200917 was a caller crashing with ESRCH reading /proc/<pid>/stat of an exiting process (its stderr in that run's log). moq-dev/quest#589 already fixed it, landing two hours after that run. 100 runs at 20-way concurrency pass here.
  • lets a phase finish its cleanup once the enclosing scope signals it: reproduced at about 1 in 300 to 500 runs under 16 to 48 concurrent test processes, logging quest: cannot signal every process in quest-gate-...scope, only its command. Instrumenting showed the systemctl --user kill helper itself was killed by SIGTERM. A gate inside a Quest scope runs its command in a sibling scope and forwards its owner's signal there, but it spawned that helper inside the enclosing scope's cgroup while the enclosing owner's own systemctl kill --kill-whom=all was still sweeping it (systemd repeats its sweep to catch new forks). The helper died of the very signal it forwarded, so the gate fell back to signalling only its command and cut the leaf's cleanup short (a missing done, or a hang).

Fix: contain signals its cgroup tree from the gate itself (which handles the signal), reading cgroup.procs recursively and repeating until a pass finds no new process, as systemd does. This drops the systemctl kill helper, Containment.signal, and populatedOr. After the fix: 0 failures in 480 runs at 48-way concurrency (previously 1 in 480 there).

Caused by

  • moq-dev/quest#585 - moved a gate inside a Quest scope into a bound sibling scope, so the systemctl kill helper from moq-dev/quest#429 started inside a cgroup that was being signalled.

Public API and wire

None. Internal to src/cli/supervise.ts.

Choices made without asking

  • Signal in-process rather than keep systemctl kill behind a trap '' TERM shell wrapper: the wrapper still leaves a window between fork and trap.
  • Reading PIDs from cgroup.procs and then calling kill has the same PID-reuse window systemd had before pidfds; Bun exposes no pidfd, and reuse within that microsecond window would need the host to cycle through every PID.
  • A failure to signal (other than ESRCH), such as EPERM from another user's process, keeps the old fallback: report it, signal only the command, and stop draining.

Follow-ups

  • Seen while stress-testing at 48-way load: is stopped when the enclosing scope's owner kills it failed once on until's fixed 5 s deadline (200 x 25 ms) in test/gate.test.ts. That is a wall-clock timeout, not a race, so it belongs to Test CPU timeout, being worked in parallel; recommend that quest's PR also covers until's deadline rather than a new quest.

Checks

  • quest gate just check: pass
  • quest gate just test: pass (1222 passed, 5 skipped)
  • Stress reproduction scripts as described above.

Models

Size: S
Implementer: Claude Code, claude-opus-5-5, low effort

🤖 Generated with Claude Code

(Written by Claude Opus 5.5 with low)

Review

Rewritten by quest review after each round.

  • Reviewer: Codex, gpt-6.1-sol, medium effort
  • Session: 01a1ca3a-8ede-7a93-97d4-aa36e75456f1, round 3
  • Head: 50ca040
  • Base: origin/main at a239765
  • Score: 10/10, The incremental diff contains only merged base changes, with no new branch changes to review and no outstanding findings.
  • Findings: none

kixelated and others added 2 commits November 11, 2026 00:53
…cope kills

A gate inside a Quest scope forwarded its owner's signal with systemctl kill,
started in the enclosing cgroup while that scope's owner was still signalling
it, so the helper could die of the signal it forwarded. Signal the cgroup tree
from the gate itself, repeating until no new process appears.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Gate tests pass deterministically fix(gate): signal a scope from the gate, not a helper the enclosing scope kills Nov 11, 2026
…gnalled

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 11, 2026 10:54
@coderabbitai

coderabbitai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Pre-merge checks: Inconclusive 2

Failed Failed checks: 2 inconclusive

Check name Status Explanation Resolution
Title check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
Description check Pending This assessment has not completed. The check will be retried on the next pre-merge check run.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated
kixelated added this pull request to the merge queue Nov 11, 2026
Merged via the queue into main with commit 9bcfd22 Nov 11, 2026
20 checks passed
@kixelated
kixelated deleted the quest/m0/gate-test-races branch November 11, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant