From 0ccf50205b502b5b7a77bcb8ed31490081e9b2a4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 9 Nov 2026 09:15:28 -0800 Subject: [PATCH 1/9] chore(quest): claim quest/m1/gate-cpu-weight.md Quest-Claim: ff6815e182fe58aa From c2b9bcf98923f48b4491935b60fd074f8769b2d6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 9 Nov 2026 09:33:30 -0800 Subject: [PATCH 2/9] feat(gate): run gated commands at low CPU priority Linux runs each gated command in a systemd scope in background.slice (CPUWeight=30), also inside a Quest scope, where the gate's scope is a sibling bound to the enclosing one instead of a child cgroup. The process group owner runs it under nice -n 10, and the Windows stub drops to below-normal priority. Drops the delegated child-cgroup path (nest). Co-Authored-By: Claude Opus 5.5 --- README.md | 22 +++--- quest/m1/gate-cpu-weight.md | 70 ----------------- src/cli/gate.ts | 50 ++++++------ src/cli/job.ts | 2 + src/cli/supervise.ts | 153 +++--------------------------------- test/gate.test.ts | 48 +++++------ 6 files changed, 75 insertions(+), 270 deletions(-) delete mode 100644 quest/m1/gate-cpu-weight.md diff --git a/README.md b/README.md index 5fa62416..81b9bbdf 100644 --- a/README.md +++ b/README.md @@ -1168,7 +1168,8 @@ gate exits with 128 plus the signal's number. How far ownership reaches depends on the host: - Linux with a systemd 254+ user manager, unified cgroup v2, and Linux 5.14+ - runs the command in its own systemd user scope, as `quest worktree run` does, + runs the command in its own systemd user scope in `background.slice`, whose + low CPU weight keeps the desktop session ahead of gated work under contention, and owns every descendant, including ones that start their own process group or session. Only a process that asks another service manager to start it escapes. @@ -1176,7 +1177,7 @@ on the host: - macOS, and Linux without such a user manager (as in most containers and on Ubuntu 22.04), run the command as the leader of a new session and own its process group. A descendant that moves to another process group or session, - as a daemon does, is outside it. The session has no controlling terminal: the + as a daemon does, is outside it. The command runs under `nice -n 10`. The session has no controlling terminal: the command keeps the terminal as its streams, but cannot open `/dev/tty`, and Ctrl-Z suspends only Quest. Nothing reports that a process group emptied, so the command gets a pipe on descriptor 3, which every descendant inherits @@ -1184,20 +1185,19 @@ on the host: start. A cancelled gate waits until every process holding it has exited, including one outside the group that the signal never reached. - Windows runs the command under a small Bun stub in a Job Object that kills - every process left in it once the stub exits, and lets none break away. It + every process left in it once the stub exits, and lets none break away. The + stub runs at below-normal priority, which everything it starts inherits. It owns every descendant, including detached ones, and stops them once the command exits or the gate is killed. Windows has no signals to forward: Ctrl-C reaches every process on the console, and killing the gate terminates the job. A nested invocation runs immediately inside the admitted command. A gate inside -another Quest scope, such as `quest worktree run`, runs its command in a child -cgroup of its own within that scope, which systemd delegates to the scope's -processes, so the gate owns its own subtree and the enclosing scope still -reaches it; a cancelled gate there waits until that child cgroup is empty. -A scope started by an older Quest, without delegation, keeps -ownership instead: a scope of the gate's own would take the command out of -the enclosing one's reach. Internal compiler or test worker parallelism remains +another Quest scope, such as `quest worktree run`, still runs its command in a +background scope of its own, so only the gated command yields the CPU, not the +rest of the enclosing scope. That scope is bound to the enclosing one, so the +enclosing owner's kill stops it too, and the gate forwards that owner's signals +to it. Internal compiler or test worker parallelism remains the command's responsibility. This repository's `just check` and `just test` run through the gate. @@ -1550,8 +1550,6 @@ is empty, and a second signal kills everything at once. A run inside another Quest scope, such as a gate's, binds its own scope to that one, so the enclosing owner's kill stops it too, and leaves a repeated signal's kill to that owner: it may be the copy of SIGTERM that Just forwards. -The scope is delegated, so a `quest gate` inside it owns its command through a -child cgroup of the scope. Work handed to another service manager, such as `systemd-run` or a container daemon, escapes the scope. If the supervisor itself is killed, its command keeps running and its scratch is kept; the next `quest worktree` command drops the run diff --git a/quest/m1/gate-cpu-weight.md b/quest/m1/gate-cpu-weight.md deleted file mode 100644 index 2cc886d2..00000000 --- a/quest/m1/gate-cpu-weight.md +++ /dev/null @@ -1,70 +0,0 @@ -# [S] Gate scopes yield the CPU to the interactive session - -## Goal - -`quest gate` keeps concurrent expensive commands from starving the -interactive desktop session. Today two full gates of a large repository -(`just check` plus `just test`, about 2,600 CPU-s each) started together on a -24-core desktop hold host load at 62.9 and CPU pressure (PSI some) at 80.6% -for about 150 s, saturating the desktop. With this quest, the same -pair still runs to completion, but the session's own processes keep CPU -priority over gated work. Admission is unchanged: this quest changes where -admitted commands run, not when they are admitted. - -## Plan - -Decided 2026-11-06 (maintainer, /quest-complete follow-up): - -- How should a gate keep the session responsive? - - ✅ Lower the gated command's CPU weight (recommended) - - Charge admission by each command's measured cores - - Both - - Admission math stays as it is; a weight change works on contention as it - happens, while admission decides from the 1-minute load average. - -Today, on Linux with a systemd user manager, `scope` in `src/cli/gate.ts` -calls `systemdScopes.run` in `src/cli/supervise.ts`, whose `systemd-run ---user --scope` names no slice or weight. Each gate scope therefore lands in -`app.slice` at the default `CPUWeight=100`, a sibling of the desktop's -applications, and `app.slice` weighs the same as `session.slice`, which holds -the GNOME session. systemd's user manager also ships `background.slice` at -`CPUWeight=30` for exactly this kind of work. Checked on 2026-11-06 with -`systemctl --user show -p Slice,CPUWeight` against running gate -scopes. - -- Run gate-admitted commands in `background.slice`, or at a low `CPUWeight`, - so the session keeps priority. Weight only matters under contention, so a - lone gate on an idle host should keep its wall time. -- Cover every Linux path a gated command takes, and lower only the gated - command, never the Quest scope around it. `quest worktree run` can hold a - whole agent session, so moving its scope into `background.slice` or - reweighting it would slow the session's non-gated work too. -- A gate inside a delegated Quest scope runs its command in a child cgroup of - that scope (`nest`). A lower weight there would not help the desktop. A - child's weight competes only with its siblings, so the gated command would - still take the whole scope's share against the desktop. cgroup v2 also lets - the scope give its children a CPU weight only once no process sits in the - scope's own cgroup, where the session runs. -- So the likely shape is a sibling scope. The gate runs its command in a scope - of its own in `background.slice`, as an unnested gate does, bound to the - enclosing scope with `BindsTo=` and `KillSignal=SIGKILL`, as - `systemdScopes.run` already binds a scope inside another. The enclosing - scope keeps its slice and weight, so the session's own work keeps its - priority and only the gated command yields. The same applies to a gate in a - Quest scope that predates delegation, which today runs its command in the - scope itself (`inside`). -- Leaving the enclosing cgroup gives up what `nest` relies on: the enclosing - scope's `cgroup.kill`, and its owner's signal to every process in it. The - implementation must show that stopping the enclosing run, its liveness check - in `worktree.ts`, and a forwarded signal still reach the gated command. If - that cannot hold, pick another shape and say why in the PR. -- Add `IOWeight` too only if the verification shows the session stalling on - disk with CPU weight alone. -- The process-group owner (Linux without systemd, macOS) and Windows have no - cgroup. Lower the command's priority there by each platform's equivalent, - or say in the PR why a platform is left as is. -- Verify with two concurrent full gates of a large repository, before and - after: the session's responsiveness under load (for example, scheduling - latency of a probe in `session.slice` or `app.slice`), and each gate's - wall time, solo and paired. Record the numbers in the PR. diff --git a/src/cli/gate.ts b/src/cli/gate.ts index f49be99f..95ea98d6 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -10,7 +10,7 @@ import type { Readable } from "node:stream"; import { setTimeout as sleep } from "node:timers/promises"; import { type Cancellation, cancelled, follow, forwards, GROUP, interrupts, received, requested } from "./interrupt"; import { stubbed } from "./job"; -import { delegated, nest, questScope, startTime, systemdScopes } from "./supervise"; +import { questScope, startTime, systemdScopes } from "./supervise"; /** * One admitted command: a unique id, the wall-clock time it was admitted, and the process ID of @@ -546,33 +546,39 @@ export const direct: Owner = plain(false); export const inside: Owner = plain(true); /** - * Run the command in its own systemd scope, whose cgroup holds every descendant (see supervise.ts). - * `owner` never picks it inside another Quest scope. + * Run the command in its own systemd scope, whose cgroup holds every descendant (see supervise.ts), + * in the user manager's `background.slice`, whose low CPU weight keeps the desktop session ahead + * of gated work under contention. Inside a Quest scope, the gate's scope is a sibling bound to + * `enclosing`, so the enclosing scope keeps its own weight for the session's other work, its + * owner's kill stops the gate's scope too, and the gate forwards that owner's signals to it. */ -export const scope: Owner = (command, env, cancel) => - systemdScopes.run(`quest-gate-${randomUUID()}.scope`, process.cwd(), command, { env, cancel, within: null }); - -/** - * Inside a delegated Quest scope, run the command in a new child of `parent`, this process's cgroup, - * which holds every descendant like a scope of its own and stays within the enclosing scope's reach. - * That scope's owner signals every process in it, so this leaves a received signal and the kill at - * its second one to that owner, as `inside` does, and lets the child's processes finish their own - * cleanup before it kills what is left. - */ -export const nested = - (parent: string): Owner => +const background = + (enclosing: string | null): Owner => (command, env, cancel) => - nest(parent, `quest-gate-${randomUUID()}`, process.cwd(), command, { env, cancel }); + systemdScopes.run(`quest-gate-${randomUUID()}.scope`, process.cwd(), command, { + env, + cancel, + within: enclosing, + slice: "background.slice", + }); + +/** Run the command in a background scope of its own, outside any Quest scope (see `background`). */ +export const scope: Owner = background(null); + +/** Run the command in a background scope bound to the Quest scope `enclosing` (see `background`). */ +export const bound = (enclosing: string): Owner => background(enclosing); /** * On Windows, run the command under a stub in a kill-on-close Job Object that forbids breakaway - * (see job.ts), so every descendant stops once the command exits or the gate stops the stub. + * (see job.ts), so every descendant stops once the command exits or the gate stops the stub. The + * stub lowers itself to below-normal priority first, which everything it starts inherits. */ export const job: Owner = (command, env, cancel) => direct(...stubbed(command, env), cancel); /** * Run the command as the leader of a new session, whose process group holds every descendant that - * does not move to another group or session. Once the command exits, the rest of its group is + * does not move to another group or session. `nice` lowers its priority, which every descendant + * inherits, since no cgroup weighs it against the interactive session. Once the command exits, the rest of its group is * killed. The session has no controlling terminal: the command keeps its inherited streams, but a * terminal's signals reach it only through Quest, which stands in for the terminal: every SIGINT, * SIGTERM, and SIGHUP Quest receives goes to the whole group, as a terminal's Ctrl-C would, and @@ -586,7 +592,7 @@ export const job: Owner = (command, env, cancel) => direct(...stubbed(command, e */ export const group: Owner = (command, env, cancel) => new Promise((resolve, reject) => { - const child = spawn(command[0], command.slice(1), { + const child = spawn("nice", ["-n", "10", ...command], { env: { ...env, [GROUP]: "1" }, stdio: ["inherit", "inherit", "inherit", "pipe"], detached: true, @@ -641,9 +647,7 @@ const TOKEN = 3; /** * The strongest owner this host offers. A nested call runs inside its admitted command, which an * enclosing Quest owner also signals when it is Quest's group or a Quest scope (see `inside`). A - * call inside a Quest scope owns a child cgroup of its own there once the scope is delegated, and - * otherwise runs inside the scope: a scope of its own would take its command out of reach of the - * enclosing one. Linux owns descendants through a systemd scope, and settles for a process group, + * call inside a Quest scope runs its command in a background scope bound to it. Linux owns descendants through a systemd scope, and settles for a process group, * like macOS, without a systemd user manager. Windows owns them through a Job Object. */ export function owner(env = process.env, cgroup = "/proc/self/cgroup"): Owner { @@ -651,7 +655,7 @@ export function owner(env = process.env, cgroup = "/proc/self/cgroup"): Owner { const enclosing = process.platform === "linux" ? questScope(cgroup) : undefined; if (env[ADMITTED] === "1") return env[GROUP] === "1" || enclosing ? inside : direct; if (process.platform === "win32") return job; - if (enclosing) return delegated(enclosing.unit) ? nested(enclosing.group) : inside; + if (enclosing) return bound(enclosing.unit); if (process.platform !== "linux") return group; try { systemdScopes.check(); diff --git a/src/cli/job.ts b/src/cli/job.ts index a7f445aa..9014d339 100644 --- a/src/cli/job.ts +++ b/src/cli/job.ts @@ -28,6 +28,8 @@ export const STUB = [ "delete process.env.QUEST_GATE_JOB;", "if (original === null) delete process.env.BUN_BE_BUN; else process.env.BUN_BE_BUN = original;", "if (process.platform === 'win32') {", + " const os = require('node:os');", + " os.setPriority(os.constants.priority.PRIORITY_BELOW_NORMAL);", " try {", " const { dlopen, FFIType, ptr } = require('bun:ffi');", " const handle = FFIType.u64;", diff --git a/src/cli/supervise.ts b/src/cli/supervise.ts index e4f9b11a..b75fa429 100644 --- a/src/cli/supervise.ts +++ b/src/cli/supervise.ts @@ -1,21 +1,9 @@ // Linux process ownership for supervised checkout commands and gated ones. Each run gets its own // systemd user scope, named once and never reused, so its cgroup holds every descendant that does -// not ask another service manager to start it. The scope is delegated, so a gate inside it owns its -// command through a child cgroup of its own (see `nest`). Other hosts fail explicitly; only the -// gate settles for a process group there (see `group` in gate.ts). +// not ask another service manager to start it. Other hosts fail explicitly; only the gate settles +// for a process group there (see `group` in gate.ts). import { type ChildProcessWithoutNullStreams, execFile, execFileSync, spawn } from "node:child_process"; -import { - closeSync, - existsSync, - constants as fs, - mkdirSync, - openSync, - readdirSync, - readFileSync, - rmdirSync, - watch, - writeSync, -} from "node:fs"; +import { closeSync, existsSync, constants as fs, openSync, readFileSync, watch, writeSync } from "node:fs"; import { constants, hostname } from "node:os"; // Cgroup paths are POSIX paths, also in the tests that parse them on Windows. import { join } from "node:path/posix"; @@ -80,6 +68,8 @@ export interface RunOptions { * owner, since it may be a copy of the one that owner delivered, as Just forwards SIGTERM. */ within?: string | null; + /** The slice the scope runs in; the user manager's default, `app.slice`, when omitted. */ + slice?: string; } export interface Scopes { @@ -268,8 +258,7 @@ export const systemdScopes: Scopes = { "--collect", `--unit=${unit}`, ...bound, - // A gate inside the scope runs its command in a child cgroup of its own (see `nest`). - "--property=Delegate=yes", + ...(options.slice ? [`--slice=${options.slice}`] : []), "--expand-environment=no", "--", "/bin/sh", @@ -282,8 +271,7 @@ export const systemdScopes: Scopes = { name: unit, unsupported: UNSUPPORTED, launcher: "systemd-run", - signal: (_group, name, done) => signalScope(unit, name, done), - inside: false, + signal: (name, done) => signalScope(unit, name, done), bound: Boolean(enclosing), }, cwd, @@ -301,15 +289,8 @@ interface Containment { unsupported: string; /** The program that starts the handshake shell. */ launcher: string; - /** Send `name` to every process in the cgroup `group`, then call `done`, with why when it could not. */ - signal(group: string, name: NodeJS.Signals, done: (error?: string) => void): void; - /** - * Whether the cgroup lies within an enclosing Quest scope, whose owner delivers each signal it - * receives to every process in the scope, this cgroup's included. A signal this process received - * then already reached them all, so it is not delivered again, and they get to finish their - * cleanup as if it had been. - */ - inside: boolean; + /** Send `name` to every process in the cgroup, then call `done`, with why when it could not. */ + signal(name: NodeJS.Signals, done: (error?: string) => void): void; /** * Whether an enclosing Quest owner kills this cgroup at its own second signal, so a repeated * signal this process receives, which may be a copy of that owner's, leaves the kill to it. @@ -354,14 +335,10 @@ function contain( interrupted = cancelled(cancel.signal); // The shell reads end-of-file and never starts the command. if (!admitted) return admission.destroy(); - if (!(describe.inside ? forwards(cancel.signal, true) : forwards(cancel.signal))) { - // The enclosing scope's owner already signalled every process here (see `inside`). - signalled = describe.inside; - return; - } + if (!forwards(cancel.signal)) return; const name = requested(cancel.signal); signalled = true; - describe.signal(group!, name, (error) => { + describe.signal(name, (error) => { // The cgroup empties once its last process exits, which leaves nothing to signal. if (!error || !group || !populatedOr(group, true)) return; process.stderr.write( @@ -444,61 +421,6 @@ function contain( }); } -/** Remove an empty cgroup and any empty child cgroups left in it; one already gone is fine. */ -function prune(group: string): void { - let entries; - try { - entries = readdirSync(group, { withFileTypes: true }); - } catch (error) { - if ((error as NodeJS.ErrnoException).code === "ENOENT") return; - throw error; - } - for (const entry of entries) if (entry.isDirectory()) prune(join(group, entry.name)); - try { - rmdirSync(group); - } catch (error) { - if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; - } -} - -/** - * Send `signal` to every process in the cgroup and its children, until a pass finds none it has not - * signalled yet, as systemd does for a unit, then call `done`. - */ -function signalGroup(group: string, signal: NodeJS.Signals, done: (error?: string) => void): void { - const sent = new Set(); - const pass = (dir: string): boolean => { - let found = false; - let entries; - try { - entries = readdirSync(dir, { withFileTypes: true }); - for (const line of readFileSync(join(dir, "cgroup.procs"), "utf8").split("\n")) { - const pid = Number(line); - if (!line || sent.has(pid)) continue; - sent.add(pid); - found = true; - try { - process.kill(pid, signal); - } catch (error) { - if ((error as NodeJS.ErrnoException).code !== "ESRCH") throw error; - } - } - } catch (error) { - // The cgroup was removed while it was read. - if (["ENOENT", "ENODEV"].includes((error as NodeJS.ErrnoException).code ?? "")) return found; - throw error; - } - for (const entry of entries) if (entry.isDirectory() && pass(join(dir, entry.name))) found = true; - return found; - }; - try { - while (pass(group)); - } catch (error) { - return done(error instanceof Error ? error.message : String(error)); - } - done(); -} - /** Matches a Quest scope's unit as a component of a cgroup path. */ const QUEST_SCOPE = /\/(quest-[a-z]+-[0-9a-f-]+\.scope)(?=\/|$)/g; @@ -514,56 +436,3 @@ export function questScope(self = "/proc/self/cgroup"): { unit: string; group: s const unit = path === undefined ? undefined : [...path.matchAll(QUEST_SCOPE)].at(-1)?.[1]; return unit === undefined ? undefined : { unit, group: join("/sys/fs/cgroup", path!) }; } - -/** - * Whether systemd delegated the scope's cgroup subtree, so its processes may create child cgroups. - * A scope from a Quest that predates delegation, or a manager that cannot be asked, is not. - */ -export function delegated(unit: string): boolean { - try { - return ( - execFileSync("systemctl", ["--user", "show", "--property=Delegate", "--value", unit], { - encoding: "utf8", - stdio: ["ignore", "pipe", "pipe"], - }).trim() === "yes" - ); - } catch { - return false; - } -} - -/** - * Run a command in a new child cgroup of `parent`, inside a delegated Quest scope, like a scope of - * its own (see `Scopes.run`): every descendant stays in the child, which the enclosing scope's - * `cgroup.kill` also reaches, and the child is removed once empty. The handshake shell moves itself - * into the child before it reports, so nothing it starts runs outside it. - */ -export async function nest( - parent: string, - name: string, - cwd: string, - command: string[], - options: RunOptions = {}, -): Promise { - const group = join(parent, name); - const unsupported = `quest gate cannot run the command in a child cgroup of its Quest scope (${group})`; - mkdirSync(group); - let status: number; - try { - status = await contain( - // Writing 0 to cgroup.procs moves the writing process, the shell itself. - ["/bin/sh", "-c", `echo 0 >"$0/cgroup.procs" || exit 125; ${handshake}`, group], - (path) => join("/sys/fs/cgroup", path) === group, - { name: group, unsupported, launcher: "/bin/sh", signal: signalGroup, inside: true, bound: true }, - cwd, - command, - options, - ); - } catch (error) { - // The cgroup may still hold processes. - if (!(error instanceof Retained)) prune(group); - throw error; - } - prune(group); - return status; -} diff --git a/test/gate.test.ts b/test/gate.test.ts index b897e8e0..5b2ec482 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -1,5 +1,4 @@ import { spawn, spawnSync } from "node:child_process"; -import { randomUUID } from "node:crypto"; import { once } from "node:events"; import { chmodSync, @@ -713,8 +712,8 @@ test("a nested call, or one inside a Quest scope, leaves its command to the encl if (process.platform !== "linux") return; expect(owner({ QUEST_GATE_ADMITTED: "1" }, gated)).toBe(inside); expect(owner({ QUEST_GATE_ADMITTED: "1" }, child)).toBe(inside); - // No such scope runs, so it is not delegated. - expect(owner({}, worktree)).toBe(inside); + // An unadmitted call runs beside the enclosing scope, not inside it. + expect([direct, inside, scope]).not.toContain(owner({}, worktree)); }); /** Gone, or a zombie its new parent has not reaped yet: either way it runs no code. */ @@ -1011,7 +1010,7 @@ async function insideScope(launch: (command: string[], env: NodeJS.ProcessEnv) = import { direct, group, inside, job, owner, scope } from ${JSON.stringify(gate)}; const dir = ${JSON.stringify(dir)}; const own = owner(process.env); - const kind = new Map([[direct, "direct"], [group, "group"], [inside, "inside"], [job, "job"], [scope, "scope"]]).get(own) ?? "nested"; + const kind = new Map([[direct, "direct"], [group, "group"], [inside, "inside"], [job, "job"], [scope, "scope"]]).get(own) ?? "bound"; let report = { kind }; if (${run}) { const never = new AbortController().signal; @@ -1032,27 +1031,30 @@ async function insideScope(launch: (command: string[], env: NodeJS.ProcessEnv) = } describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () => { - test("owns a child cgroup of a delegated scope and removes it", { timeout: 30_000 }, async () => { + test("runs its command in a background scope beside the enclosing one", { timeout: 30_000 }, async () => { const report = await insideScope((command, env) => scope(command, env, cancellation().cancel), true); - expect(report).toMatchObject({ kind: "nested", status: 7, stopped: true, removed: true }); - // The child sits inside the enclosing scope, so that scope's cgroup.kill still reaches it. - expect(report.path).toMatch(/\/quest-gate-[0-9a-f-]+\.scope\/quest-gate-[0-9a-f-]+$/); + expect(report).toMatchObject({ kind: "bound", status: 7, stopped: true, removed: true }); + // A sibling, so only the gated command yields the CPU, not the rest of the enclosing scope. + expect(report.path).toMatch(/\/background\.slice\/quest-gate-[0-9a-f-]+\.scope$/); }); - test("leaves its command to a scope that is not delegated", { timeout: 30_000 }, async () => { - // A scope started as a Quest before delegation would have. - const unit = `quest-worktree-${randomUUID()}.scope`; - const launch = (command: string[], env: NodeJS.ProcessEnv) => - new Promise((resolve, reject) => { - const child = spawn( - "systemd-run", - ["--user", "--scope", "--quiet", "--collect", `--unit=${unit}`, "--", ...command], - { env, stdio: "inherit" }, - ); - child.once("error", reject); - child.once("exit", (code) => resolve(code ?? 1)); - }); - expect(await insideScope(launch, false)).toEqual({ kind: "inside" }); + test("is stopped when the enclosing scope's owner kills it", { timeout: 30_000 }, async () => { + const dir = tempDir(); + const ready = join(dir, "ready"); + const pid = join(dir, "pid"); + // Ignoring SIGTERM, it outlives the forwarded signal, so only the bound scope's stop ends it. + const command = [ + "/bin/sh", + "-c", + `trap '' TERM; sleep 1000 & echo $! > ${quote(pid)}; : > ${quote(ready)}; wait`, + ]; + const { stop, kill, cancel } = cancellation(); + const result = scope(nestedGate(dir, command), outside(), cancel); + await until(() => existsSync(ready), "the command to start"); + stop.abort("SIGTERM"); + kill.abort(); + await result; + await until(() => stopped(Number(readFileSync(pid, "utf8"))), "the gated command to stop"); }); /** @@ -1108,7 +1110,7 @@ describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () = expect(existsSync(leaf.done)).toBe(true); }); - test("cancelled by its caller, signals every process in its child cgroup", { timeout: 30_000 }, async () => { + test("cancelled by its caller, signals every process in its scope", { timeout: 30_000 }, async () => { const dir = tempDir(); const leaf = signalled(dir, ["SIGTERM"], 200); const abort = join(dir, "abort"); From 34a0aad1a7d4a46306a3f07248377700699e9735 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 9 Nov 2026 09:35:41 -0800 Subject: [PATCH 3/9] fix(supervise): confirm a scope stopped only once its bound gate scopes are empty Co-Authored-By: Claude Opus 5.5 --- src/cli/supervise.ts | 34 ++++++++++++++++++++++++++++++---- test/gate.test.ts | 3 ++- 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/src/cli/supervise.ts b/src/cli/supervise.ts index b75fa429..66932760 100644 --- a/src/cli/supervise.ts +++ b/src/cli/supervise.ts @@ -214,6 +214,26 @@ function scopeGroup(unit: string): string | undefined { return join("/sys/fs/cgroup", group); } +/** + * The gate scopes bound to `unit` (see `RunOptions.within`). systemd stops them once `unit` stops, + * but only after its cgroup empties, so whoever confirms `unit` stopped must wait for them too. + */ +function boundTo(unit: string): string[] { + const show = execFileSync("systemctl", ["--user", "show", "--property=Id,BindsTo", "quest-gate-*.scope"], { + encoding: "utf8", + stdio: ["ignore", "pipe", "pipe"], + }); + return show + .split("\n\n") + .filter((entry) => /^BindsTo=(.*)$/m.exec(entry)?.[1].split(" ").includes(unit)) + .map((entry) => /^Id=(.*)$/m.exec(entry)![1]); +} + +/** Kill every gate scope bound to `unit` and resolve once each is empty. */ +async function stopBound(unit: string): Promise { + for (const bound of boundTo(unit)) await systemdScopes.stop(bound); +} + export const systemdScopes: Scopes = { check() { if (process.platform !== "linux" || !killable()) throw new Error(UNSUPPORTED); @@ -235,14 +255,16 @@ export const systemdScopes: Scopes = { live(unit) { const group = scopeGroup(unit); - return group !== undefined && populated(group); + return (group !== undefined && populated(group)) || boundTo(unit).some((bound) => systemdScopes.live(bound)); }, async stop(unit) { const group = scopeGroup(unit); - if (group === undefined) return; - kill(group); - await vacated(group); + if (group !== undefined) { + kill(group); + await vacated(group); + } + await stopBound(unit); }, run(unit, cwd, command, options = {}) { @@ -272,6 +294,7 @@ export const systemdScopes: Scopes = { unsupported: UNSUPPORTED, launcher: "systemd-run", signal: (name, done) => signalScope(unit, name, done), + emptied: () => stopBound(unit), bound: Boolean(enclosing), }, cwd, @@ -291,6 +314,8 @@ interface Containment { launcher: string; /** Send `name` to every process in the cgroup, then call `done`, with why when it could not. */ signal(name: NodeJS.Signals, done: (error?: string) => void): void; + /** Stop what the cgroup's end leaves to systemd, once the cgroup is empty. */ + emptied(): Promise; /** * Whether an enclosing Quest owner kills this cgroup at its own second signal, so a repeated * signal this process receives, which may be a copy of that owner's, leaves the kill to it. @@ -400,6 +425,7 @@ function contain( if (signalled) await vacated(group, drain.signal); kill(group); await vacated(group); + await describe.emptied(); } catch (error) { throw new Retained( `cannot confirm that ${describe.name} stopped: ${error instanceof Error ? error.message : error}`, diff --git a/test/gate.test.ts b/test/gate.test.ts index 5b2ec482..18b8ffcd 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -1054,7 +1054,8 @@ describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () = stop.abort("SIGTERM"); kill.abort(); await result; - await until(() => stopped(Number(readFileSync(pid, "utf8"))), "the gated command to stop"); + // The enclosing run resolves only once the bound scope is empty too. + expect(stopped(Number(readFileSync(pid, "utf8")))).toBe(true); }); /** From e66954583348a394ccc32882135bf31d6f9de1b3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 9 Nov 2026 22:24:13 -0800 Subject: [PATCH 4/9] chore(quest): claim quest/m1/gate-fifo-admission.md Quest-Claim: c303c1a339dd80d9 From cc14fa7381773c0d3c53be4ad595febcf00040f1 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 00:28:39 -0800 Subject: [PATCH 5/9] feat(gate): admit waiting commands first come, first served Each waiting caller records its place in the admission ledger and counts every caller queued ahead of it as a recent admission, so a later caller starts first only while the host has room for all of them, and only the head of the queue waits out the spacing. Priority and host callers go ahead of the queue; a quick caller still starts at its deadline. Co-Authored-By: Claude Opus 5.5 --- README.md | 23 ++++-- quest/m1/gate-fifo-admission.md | 44 ----------- src/cli/gate.ts | 130 +++++++++++++++++++++++--------- test/gate.test.ts | 102 ++++++++++++++++++++++++- 4 files changed, 215 insertions(+), 84 deletions(-) delete mode 100644 quest/m1/gate-fifo-admission.md diff --git a/README.md b/README.md index 1ef9d14c..c4c8da22 100644 --- a/README.md +++ b/README.md @@ -1114,6 +1114,14 @@ host has room even counting those, commands start at once: a burst of gated test runs on an idle host starts together. Once only the raw load and memory have room, commands start one at a time, 5 seconds after the previous admission. +Waiting commands start first come, first served. Each records its place in a +queue when it starts waiting, and counts every command queued ahead of it as +one more recent admission: it starts before them only while the host has room +for all of them, and only the first in line waits out the 5-second spacing. A +command's wait is therefore bounded by the work queued ahead of it, never by +commands that started waiting later. A place is freed once its command starts, +its gate is cancelled, or its gate process exits. + The load limit applies only while a gated command is still running. Otherwise the load comes from work the gate cannot wait out, so commands start one at a time as if the raw load had room. A freshly booted hosted macOS runner is one @@ -1128,11 +1136,13 @@ an account missing from it, as in arbitrary-UID containers, uses its UID can write it. Two kinds of command go ahead of the rest. `quest gate --quick ` waits -at most a minute, then starts the command whatever the host's load and memory; +at most a minute, then starts the command whatever the host's load, memory, and +place in the queue; pass it only for cheap commands, such as a formatter or `quest check`, that cost less to run than to hold back. A check that lands PRs goes first: with `QUEST_GATE_PRIORITY=1` in its environment, every gate inside the check waits -for memory but never for load, whether or not other gated commands run. Both +for memory but never for load or the queue, whether or not other gated commands +run. Both record their admissions, so the other callers count them against the host's headroom. Set the variable where the check starts, such as a merge queue build's workflow or the land daemon's sandbox, rather than relying on a slot in @@ -1153,7 +1163,10 @@ where a reused process ID could pass for a holder (see below), refuses Each admission records the gate's process ID and, except on Windows, when the gate started, and counts as running until that process exits; a reused process ID counts as running only on Windows, where it makes a command wait for the -load again. The gate holds no lock while the command runs, and it owns the +load again. A place in the queue records the same, except that on Windows the +waiting gate refreshes it, and it frees a minute after the last refresh, so a +reused process ID cannot hold the queue. The gate holds no lock while the +command runs, and it owns the command's descendants. Once the command exits, any descendant still running is killed, so a build or test run leaves nothing behind. @@ -1215,8 +1228,8 @@ first one as Just's copy, not as a second signal: a SIGTERM to Just's process group reaches the gate both directly and through Just. A repeated SIGTERM still kills, though when sent to Just or the gate alone, or when the two copies arrive together, it takes a third; a repeated Ctrl-C, which Just does not -forward, kills at the second. There is no fairness guarantee, and no timeout except -`--quick`'s: any other command waits as long as the host stays busy. +forward, kills at the second. There is no timeout except `--quick`'s: any other +command waits its turn for as long as the host stays busy. ## Route work by size diff --git a/quest/m1/gate-fifo-admission.md b/quest/m1/gate-fifo-admission.md deleted file mode 100644 index 701d92f0..00000000 --- a/quest/m1/gate-fifo-admission.md +++ /dev/null @@ -1,44 +0,0 @@ -# [M] Gate admission is first come, first served - -## Goal - -`quest gate` admits waiting commands in arrival order, so no caller is passed -by commands that started waiting after it, and its wait is bounded by the work -queued ahead of it. The host's load and memory still decide when the head of -the queue starts. `quick`, `priority`, and `host` keep their meanings. - -Today admission has no queue: every waiter polls the 1-minute load average and -available memory, and whichever sees room first starts. With about 12 agents -gating on one desktop, a diff-scoped `just check` that ran in 84 s waited 77 -minutes to be admitted -([moq-dev/moq.pro#3751](https://github.com/moq-dev/moq.pro/pull/3751)). - -## Plan - -Decided 2026-11-09 (maintainer, `/quest-plan` of moq-dev/moq.pro#3751's -follow-up): - -- Plan it: ✅ in m1, beside [gate-cpu-weight](/quest/m1/gate-cpu-weight.md) - (rejected: m0; drop). -- Mechanism: ✅ FIFO admission: waiters start in arrival order, with no cost - estimate (rejected: shortest job first from measured costs, which needs a - cost model and can starve full gates; a slot reserved for small commands; - leaving the choice to the implementer). - -The admission ledger (`src/cli/gate.ts`) already records each admission under -an atomic versioned publish; the queue can live in the same ledger, recording -each waiter when it starts waiting. A waiter whose gate process died leaves -the queue the way a dead admission leaves the ledger today. `quick` callers -still start after `QUICK_MS` whatever their place, and a `host` reservation -still holds everyone back. A `priority` caller, which today ignores load so it -goes first while the host is loaded, keeps going ahead of the queue rather than -taking a place in it. - -`admit` drops a ledger entry once it is older than `RECENT_MS` and its process -no longer runs; a queued waiter's entry must stay while its gate lives, however -long it waits, and must not count as an admission against anyone's headroom. -`admit`'s doc comment, which today promises no fairness, states the new order. - -## Related - -- [Gate scopes yield the CPU to the interactive session](/quest/m1/gate-cpu-weight.md) - changes where admitted commands run; this changes when they are admitted diff --git a/src/cli/gate.ts b/src/cli/gate.ts index 7020d382..17b8d2c3 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -16,17 +16,20 @@ import { delegated, nest, questScope, startTime, systemdScopes } from "./supervi * One admitted command: a unique id, the wall-clock time it was admitted, and the process ID of * the gate that runs it, which holds a slot until that command ends, with when that gate started * (see `processStart`), so a reused process ID holds nothing. `host` marks a command that holds the - * whole host (see `Precedence`), from when it starts waiting until it ends. + * whole host (see `Precedence`), from when it starts waiting until it ends. `queued` marks a caller + * still waiting for its turn instead, as of `at`: it holds a place in the queue, the ledger's order + * of queued entries, and no slot. */ export interface Admission { id: string; at: number; pid?: number; host?: boolean; + queued?: boolean; start?: string; } -/** One version of the host's admission ledger. Versions only grow; 0 is the empty ledger. */ +/** One version of the host's admission ledger and queue. Versions only grow; 0 is the empty ledger. */ export interface Ledger { version: number; admissions: Admission[]; @@ -59,7 +62,7 @@ export interface Limits { memory: number; } -export type Waiting = "load" | "memory" | "host" | "commands"; +export type Waiting = "load" | "memory" | "host" | "commands" | "queue"; /** * Admissions that only fit the host's raw load and memory, not its headroom, @@ -256,6 +259,7 @@ function parse(text: string): Admission[] { Number.isFinite(entry?.at) && (entry.pid === undefined || (Number.isSafeInteger(entry.pid) && entry.pid > 0)) && (entry.host === undefined || entry.host === true) && + (entry.queued === undefined || entry.queued === true) && (entry.start === undefined || typeof entry.start === "string"), ); return valid ? (value as Admission[]) : []; @@ -409,14 +413,23 @@ export function fileHost(directory: string, now = Date.now): Host { * one, the load is work the gate cannot wait out, as on a freshly booted macOS * runner, so the raw load counts as having room. * + * Callers are admitted first come, first served: each records its place in the + * queue when it starts waiting, and counts every caller queued ahead of it as + * one more recent admission, so it starts before them only while the host has + * room for them all. Only the head of the queue waits out the spacing. A place + * lasts until its caller is admitted, cancelled, or its gate ends; one whose + * gate's start time is unknown, as on Windows, could pass for a reused process + * ID, so its caller refreshes it, and it frees a minute after the last refresh. + * * A running command that holds the host keeps everyone else waiting; one that - * waits to hold it ignores load and memory, and waits only for the gated - * commands still running to end (see `Precedence`). + * waits to hold it ignores load, memory, and the queue, and waits only for the + * gated commands still running to end. A priority caller goes ahead of the + * queue, and a quick one starts at its deadline whatever its place (see + * `Precedence`). * - * Each caller decides against the newest ledger and publishes its admission - * atomically, so concurrent callers never exceed the rule and one always wins - * while it has room. No fairness promise, and no admission timeout except a - * quick caller's (see `Precedence`). + * Each caller decides against the newest ledger and publishes atomically, so + * concurrent callers never exceed the rule and one always wins while it has + * room. No admission timeout except a quick caller's. */ export async function admit( limits: Limits, @@ -426,18 +439,30 @@ export async function admit( precedence: Precedence = {}, ) { const id = randomUUID(); + // A place in the queue has an id of its own, so publishing the admission adds a new entry. + const place = randomUUID(); // Every admission names its gate's start, so a reused process ID holds no slot, as for a host holder. const self = { pid: process.pid, start: processStart(process.pid) }; const deadline = precedence.quick ? host.now() + QUICK_MS : Infinity; - for (;;) { - signal.throwIfAborted(); + const queues = !precedence.priority && !precedence.host; + + /** One decision against the newest ledger: true once admitted. */ + const turn = async (): Promise => { const ledger = host.ledger(); // Sampled after the read, so only a backward clock step dates an admission in the future. const now = host.now(); - // A running admission stays in the ledger past the minute, so later callers still see its slot. - const running = new Set(ledger.admissions.filter((admission) => host.running(admission))); - const kept = ledger.admissions.filter((admission) => admission.at > now - RECENT_MS || running.has(admission)); - const recent = kept.filter((admission) => admission.at > now - RECENT_MS); + const live = new Set(ledger.admissions.filter((admission) => host.running(admission))); + const kept = ledger.admissions.filter((admission) => + admission.queued + ? // A place whose gate cannot be told from a reused process ID lasts only while refreshed. + admission.id === place || + (live.has(admission) && (admission.start !== undefined || admission.at > now - RECENT_MS)) + : // A running admission stays in the ledger past the minute, so later callers still see its slot. + admission.at > now - RECENT_MS || live.has(admission), + ); + const admissions = kept.filter((admission) => !admission.queued); + const running = new Set(admissions.filter((admission) => live.has(admission))); + const recent = admissions.filter((admission) => admission.at > now - RECENT_MS); if (kept.some((admission) => admission.at > now)) { // Date those now, so they still count and space admissions but expire a minute after // the step. A forward step can only expire admissions up to a minute early. @@ -445,30 +470,42 @@ export async function admit( ledger, kept.map((admission) => ({ ...admission, at: Math.min(admission.at, now) })), ); - continue; + return false; } + const position = kept.findIndex((admission) => admission.id === place); + const mine = kept[position]; + if (queues && (mine === undefined || (mine.start === undefined && mine.at <= now - RECENT_MS / 2))) { + // Take a place at the back, or refresh one that would otherwise expire where it stands. + const entry = { id: place, at: now, ...self, queued: true }; + host.publish( + ledger, + mine ? kept.map((admission) => (admission === mine ? entry : admission)) : [...kept, entry], + ); + return false; + } + const ahead = kept.slice(0, Math.max(position, 0)).filter((admission) => admission.queued).length; + // An admission takes the place of this caller's entry in the queue. + const admitted = (at: number) => + host.publish(ledger, [...kept.filter((admission) => admission.id !== place), { id, at, ...self }]); // A command holding the host keeps every other caller out until it ends, quick or not. if (kept.some((admission) => admission.host && admission.id !== id && running.has(admission))) { waiting?.("host"); await host.sleep(SETTLE_MS, signal); - continue; + return false; } if (precedence.host) { // Reserve the host first, so no other caller is admitted while this one waits for the rest. if (!kept.some((admission) => admission.id === id)) host.publish(ledger, [...kept, { id, at: now, ...self, host: true }]); - else if (!kept.some((admission) => admission.id !== id && running.has(admission))) return; + else if (!kept.some((admission) => admission.id !== id && running.has(admission))) return true; else { waiting?.("commands"); await host.sleep(SETTLE_MS, signal); } - continue; - } - if (now >= deadline) { - // Recorded like any admission, so it still counts against the others' headroom. - if (host.publish(ledger, [...kept, { id, at: now, ...self }])) return; - continue; + return false; } + // Recorded like any admission, so it still counts against the others' headroom. + if (now >= deadline) return admitted(now); // Priority ignores load, whether or not a gated command runs; memory still applies. const load = precedence.priority ? 0 : running.size > 0 ? host.load() : Math.min(host.load(), limits.load); const memory = host.memory(); @@ -476,19 +513,43 @@ export async function admit( if (load > limits.load) waiting?.("load"); else if (memory < limits.memory) waiting?.("memory"); else { - const headroom = - load + recent.length < limits.load && memory - recent.length * limits.memory >= limits.memory; - const last = Math.max(-Infinity, ...recent.map((admission) => admission.at)); - wait = headroom ? 0 : last + SETTLE_MS - now; - if (wait <= 0) { - // A caller descheduled since sampling would record a stale time, which lets the - // next one skip its spacing; decide again once the stall reaches that interval. - const at = host.now(); - if (at - now < SETTLE_MS && host.publish(ledger, [...kept, { id, at, ...self }])) return; - continue; + // Every caller queued ahead takes a slot before this one, so it starts first only while + // the host has room for them all. + const taken = recent.length + ahead; + const headroom = load + taken < limits.load && memory - taken * limits.memory >= limits.memory; + if (!headroom && ahead > 0) waiting?.("queue"); + else { + const last = Math.max(-Infinity, ...recent.map((admission) => admission.at)); + wait = headroom ? 0 : last + SETTLE_MS - now; + if (wait <= 0) { + // A caller descheduled since sampling would record a stale time, which lets the + // next one skip its spacing; decide again once the stall reaches that interval. + const at = host.now(); + return at - now < SETTLE_MS && admitted(at); + } } } await host.sleep(Math.min(wait, deadline - now), signal); + return false; + }; + + try { + for (;;) { + signal.throwIfAborted(); + if (await turn()) return; + } + } finally { + // A cancelled caller leaves the queue at once, rather than once its gate ends. + for (let ledger = host.ledger(); ledger.admissions.some((admission) => admission.id === place);) { + if ( + host.publish( + ledger, + ledger.admissions.filter((admission) => admission.id !== place), + ) + ) + break; + ledger = host.ledger(); + } } } @@ -666,6 +727,7 @@ const notices: Record = { memory: "quest gate: waiting for free memory", host: "quest gate: waiting for a command holding the whole host to end", commands: "quest gate: holding the host, waiting for gated commands to end", + queue: "quest gate: waiting for commands queued earlier to start", }; /** Run from the CLI. */ diff --git a/test/gate.test.ts b/test/gate.test.ts index 7aa7def6..bedc1099 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -334,6 +334,106 @@ test("a caller holding the host reserves it, waits out gated commands, and holds expect(waited).toEqual([]); }); +test("waiting callers start in arrival order, a later one first only while the host has room for both", async () => { + const limits = { load: 4, memory: GiB }; + // A build still runs, and an earlier caller waits for its turn. + const build = { id: "build", at: -10 * RECENT_MS, ...me }; + const earlier = { id: "earlier", at: 0, ...me, queued: true }; + const directory = tempDir(); + const state = { now: 0, load: 3, memory: Infinity }; + const events: string[] = []; + const host = testHost(directory, state, events); + expect(host.publish(host.ledger(), [build, earlier])).toBe(true); + const sleep = host.sleep; + host.sleep = async (ms, signal) => { + await sleep(ms, signal); + // The earlier caller takes its turn. + const ledger = host.ledger(); + const rest = ledger.admissions.filter(({ id }) => id !== "earlier"); + if (rest.length < ledger.admissions.length) + expect(host.publish(ledger, [...rest, { id: "started", at: state.now, ...me }])).toBe(true); + }; + // Alone, this caller would start at once; behind the earlier one, it waits, then spaces itself. + await admit(limits, host, signal, (reason) => events.push(reason)); + expect(events).toEqual(["queue", `sleep ${SETTLE_MS}`, `sleep ${SETTLE_MS}`]); + expect(host.ledger().admissions).toEqual([ + build, + { id: "started", at: SETTLE_MS, ...me }, + { id: expect.any(String), at: 2 * SETTLE_MS, ...me }, + ]); + // With room for both, it starts at once and leaves the earlier caller's place. + const roomy = testHost(tempDir(), { now: 0, load: 2, memory: Infinity }, events); + expect(roomy.publish(roomy.ledger(), [build, earlier])).toBe(true); + events.length = 0; + await admit(limits, roomy, signal, (reason) => events.push(reason)); + expect(events).toEqual([]); + expect(roomy.ledger().admissions).toEqual([build, earlier, { id: expect.any(String), at: 0, ...me }]); +}); + +test("a place frees once its gate ends or its caller cancels, or a minute after its last refresh without a start", async () => { + const limits = { load: 4, memory: GiB }; + const build = { id: "build", at: -10 * RECENT_MS, ...me }; + const exited = spawnSync(process.execPath, ["-e", ""]).pid; + const directory = tempDir(); + const state = { now: 0, load: 3, memory: Infinity }; + const host = testHost(directory, state); + expect( + host.publish(host.ledger(), [ + build, + { id: "ended", at: 0, pid: exited, queued: true }, + { id: "reused", at: 0, pid: process.pid, start: "an earlier process", queued: true }, + { id: "unrefreshed", at: -RECENT_MS, pid: process.pid, queued: true }, + ]), + ).toBe(true); + const events: string[] = []; + await admit(limits, testHost(directory, state, events), signal, (reason) => events.push(reason)); + expect(events).toEqual([]); + expect(host.ledger().admissions).toEqual([build, { id: expect.any(String), at: 0, ...me }]); + // A place refreshed within the minute still holds, and a cancelled caller behind it leaves the queue. + const fresh = { id: "fresh", at: -RECENT_MS / 2, pid: process.pid, queued: true }; + const queued = testHost(tempDir(), state); + expect(queued.publish(queued.ledger(), [build, fresh])).toBe(true); + const controller = new AbortController(); + const waited: string[] = []; + queued.sleep = async () => { + expect(queued.ledger().admissions).toEqual([build, fresh, expect.objectContaining({ queued: true, ...me })]); + controller.abort(new Error("cancelled")); + }; + await expect(admit(limits, queued, controller.signal, (reason) => waited.push(reason))).rejects.toThrow( + "cancelled", + ); + expect(waited).toEqual(["queue"]); + expect(queued.ledger().admissions).toEqual([build, fresh]); +}); + +test("priority and host callers go ahead of the queue, and a quick one starts at its deadline whatever its place", async () => { + const limits = { load: 4, memory: GiB }; + const build = { id: "build", at: -10 * RECENT_MS, ...me }; + const earlier = { id: "earlier", at: 0, ...me, queued: true }; + const directory = tempDir(); + const state = { now: 0, load: 3, memory: Infinity }; + const host = testHost(directory, state); + expect(host.publish(host.ledger(), [build, earlier])).toBe(true); + const events: string[] = []; + await admit(limits, testHost(directory, state, events), signal, undefined, { priority: true }); + expect(events).toEqual([]); + await admit(limits, testHost(directory, state, events), signal, undefined, { quick: true }); + expect(events).toEqual(Array.from({ length: QUICK_MS / SETTLE_MS }, () => `sleep ${SETTLE_MS}`)); + expect(host.ledger().admissions).toEqual([ + build, + earlier, + { id: expect.any(String), at: 0, ...me }, + { id: expect.any(String), at: QUICK_MS, ...me }, + ]); + // A waiting caller runs no command, so a host holder starts at once ahead of it. + const held = testHost(tempDir(), { now: 0, load: 99, memory: 0 }, events); + expect(held.publish(held.ledger(), [earlier])).toBe(true); + events.length = 0; + await admit(limits, held, signal, undefined, { host: true }); + expect(events).toEqual([]); + expect(held.ledger().admissions).toEqual([earlier, { id: expect.any(String), at: 0, ...me, host: true }]); +}); + test("a lone quick or priority caller starts at once on load from outside the gate", async () => { // No gated command runs, so the load is work the gate cannot wait out, whatever the precedence. const state = { now: 0, load: 12, memory: Infinity }; @@ -421,7 +521,7 @@ test("a caller that dies mid-admission blocks no one past a minute", async () => await admit({ load: 1, memory: 1 }, testHost(directory, state, events), signal); expect(events).toEqual([]); // The stale draft is gone; one still being written stays. - expect(readdirSync(directory).sort()).toEqual(["2", "live.tmp"]); + expect(readdirSync(directory).sort()).toEqual(["3", "live.tmp"]); // A version cut short, as by power loss, reads as empty rather than wedging every caller. writeFileSync(join(directory, "100"), ""); expect(host.ledger()).toEqual({ version: 100, admissions: [] }); From be6b4bc3746e003b83e4837cb40642b872205879 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 08:30:53 -0800 Subject: [PATCH 6/9] quest: drop gate-fifo-admission's links to the finished gate-cpu-weight Co-Authored-By: Claude Opus 5.5 --- quest/m1/gate-fifo-admission.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/quest/m1/gate-fifo-admission.md b/quest/m1/gate-fifo-admission.md index 701d92f0..a4c57c10 100644 --- a/quest/m1/gate-fifo-admission.md +++ b/quest/m1/gate-fifo-admission.md @@ -18,8 +18,8 @@ minutes to be admitted Decided 2026-11-09 (maintainer, `/quest-plan` of moq-dev/moq.pro#3751's follow-up): -- Plan it: ✅ in m1, beside [gate-cpu-weight](/quest/m1/gate-cpu-weight.md) - (rejected: m0; drop). +- Plan it: ✅ in m1, beside gate-cpu-weight, which yields gated commands' + CPU to the interactive session (rejected: m0; drop). - Mechanism: ✅ FIFO admission: waiters start in arrival order, with no cost estimate (rejected: shortest job first from measured costs, which needs a cost model and can starve full gates; a slot reserved for small commands; @@ -38,7 +38,3 @@ taking a place in it. no longer runs; a queued waiter's entry must stay while its gate lives, however long it waits, and must not count as an admission against anyone's headroom. `admit`'s doc comment, which today promises no fairness, states the new order. - -## Related - -- [Gate scopes yield the CPU to the interactive session](/quest/m1/gate-cpu-weight.md) - changes where admitted commands run; this changes when they are admitted From 8b1c6586f25d5949cb0a741a6d42255111e514c6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 09:47:52 -0800 Subject: [PATCH 7/9] feat(gate): count running commands at their measured CPU peak The gate admitted whenever the 1-minute load average fell to the host's CPU count, so a lull between a running build's phases let another whole-host command in. Each gate that owns its command through a systemd scope now records the most CPUs the command kept busy over any 5-second sample in the last minute, and admission counts the load as at least the sum of those records, admitting nothing while they fill every CPU. Co-Authored-By: Claude Opus 5.5 --- README.md | 8 +++ src/cli/gate.ts | 140 +++++++++++++++++++++++++++++++------- src/cli/main.ts | 4 +- src/cli/supervise.ts | 2 +- test/gate.test.ts | 124 +++++++++++++++++++++++++++------ test/worktree.run.test.ts | 3 +- 6 files changed, 232 insertions(+), 49 deletions(-) diff --git a/README.md b/README.md index b6c69e57..98f3ba7f 100644 --- a/README.md +++ b/README.md @@ -1128,6 +1128,14 @@ time as if the raw load had room. A freshly booted hosted macOS runner is one such host: work outside the job holds its load above its CPU count for the whole job. Available memory is still required. +A running command's load can lull, as between a build's phases, though it will +soon take the whole host again. So where a systemd scope owns the command, its +gate samples the scope's CPU use every 5 seconds and records the most CPUs the +command kept busy over one sample in the last minute. While gated commands run, +the load counts as at least the sum of those records, and nothing starts while +they kept every CPU busy. macOS, Windows, and Linux without a systemd user +manager have no scope to measure, so the load average alone decides there. + A user's admissions on one host are recorded under `~/.cache/quest/admissions/`, whatever the caller's environment or login session. The home directory comes from the user database, not `$HOME`; diff --git a/src/cli/gate.ts b/src/cli/gate.ts index 172f4b7a..812fc6de 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -10,14 +10,15 @@ import type { Readable } from "node:stream"; import { setTimeout as sleep } from "node:timers/promises"; import { type Cancellation, cancelled, follow, forwards, GROUP, interrupts, received, requested } from "./interrupt"; import { stubbed } from "./job"; -import { questScope, startTime, systemdScopes } from "./supervise"; +import { questScope, scopeGroup, startTime, systemdScopes } from "./supervise"; /** * One admitted command: a unique id, the wall-clock time it was admitted, and the process ID of * the gate that runs it, which holds a slot until that command ends, with when that gate started * (see `processStart`), so a reused process ID holds nothing. `queued` marks a caller still * waiting for its turn instead, as of `at`: it holds a place in the queue, the ledger's order of - * queued entries, and no slot. + * queued entries, and no slot. `cores` is the most CPUs the command kept busy over any + * SETTLE_MS of the last RECENT_MS, where its gate measures it (see `measure`). */ export interface Admission { id: string; @@ -25,6 +26,7 @@ export interface Admission { pid?: number; queued?: boolean; start?: string; + cores?: number; } /** One version of the host's admission ledger and queue. Versions only grow; 0 is the empty ledger. */ @@ -51,6 +53,8 @@ export interface Host { */ publish(base: Ledger, admissions: Admission[]): boolean; sleep(ms: number, signal: AbortSignal): Promise; + /** CPU microseconds the processes in systemd scope `unit` have used, or undefined while it does not exist. */ + usage(unit: string): number | undefined; } export interface Limits { @@ -252,7 +256,8 @@ function parse(text: string): Admission[] { Number.isFinite(entry?.at) && (entry.pid === undefined || (Number.isSafeInteger(entry.pid) && entry.pid > 0)) && (entry.queued === undefined || entry.queued === true) && - (entry.start === undefined || typeof entry.start === "string"), + (entry.start === undefined || typeof entry.start === "string") && + (entry.cores === undefined || (Number.isSafeInteger(entry.cores) && entry.cores >= 0)), ); return valid ? (value as Admission[]) : []; } @@ -311,6 +316,8 @@ function remove(file: string) { */ export function fileHost(directory: string, now = Date.now): Host { const path = (version: number) => join(directory, String(version)); + // A scope's cgroup never moves, so systemd is asked for it once. + const groups = new Map(); const latest = (): number => { let names: string[]; try { @@ -387,6 +394,22 @@ export function fileHost(directory: string, now = Date.now): Host { return true; }, sleep: (ms, signal) => sleep(ms, undefined, { signal }), + usage(unit) { + const group = groups.get(unit) ?? scopeGroup(unit); + if (group === undefined) return undefined; + groups.set(unit, group); + let stat: string; + try { + stat = readFileSync(join(group, "cpu.stat"), "utf8"); + } catch (error) { + // The scope ended, and systemd removed its cgroup. + if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined; + throw error; + } + const usage = stat.match(/^usage_usec (\d+)$/m)?.[1]; + if (usage === undefined) throw new Error(`gate cannot read the CPU use of ${unit}`); + return Number(usage); + }, }; } @@ -405,6 +428,12 @@ export function fileHost(directory: string, now = Date.now): Host { * one, the load is work the gate cannot wait out, as on a freshly booted macOS * runner, so the raw load counts as having room. * + * A running command's load can lull, as between a build's phases, while it will + * soon take the whole host again. So where gates measure their commands (see + * `measure`), the load counts as at least the CPUs those commands kept busy at + * their busiest in the last minute, and nothing is admitted while they kept + * every CPU busy. + * * Callers are admitted first come, first served: each records its place in the * queue when it starts waiting, and counts every caller queued ahead of it as * one more recent admission, so it starts before them only while the host has @@ -417,7 +446,8 @@ export function fileHost(directory: string, now = Date.now): Host { * * Each caller decides against the newest ledger and publishes atomically, so * concurrent callers never exceed the rule and one always wins while it has - * room. No admission timeout except a quick caller's. + * room. No admission timeout except a quick caller's. Resolves with the + * admission's id. */ export async function admit( limits: Limits, @@ -425,7 +455,7 @@ export async function admit( signal: AbortSignal, waiting?: (reason: Waiting) => void, precedence: Precedence = {}, -) { +): Promise { const id = randomUUID(); // A place in the queue has an id of its own, so publishing the admission adds a new entry. const place = randomUUID(); @@ -480,10 +510,15 @@ export async function admit( // Recorded like any admission, so it still counts against the others' headroom. if (now >= deadline) return admitted(now); // Priority ignores load, whether or not a gated command runs; memory still applies. - const load = precedence.priority ? 0 : running.size > 0 ? host.load() : Math.min(host.load(), limits.load); + const used = precedence.priority ? 0 : [...running].reduce((sum, admission) => sum + (admission.cores ?? 0), 0); + const load = precedence.priority + ? 0 + : running.size > 0 + ? Math.max(host.load(), used) + : Math.min(host.load(), limits.load); const memory = host.memory(); let wait = SETTLE_MS; - if (load > limits.load) waiting?.("load"); + if (load > limits.load || used >= limits.load) waiting?.("load"); else if (memory < limits.memory) waiting?.("memory"); else { // Every caller queued ahead takes a slot before this one, so it starts first only while @@ -509,7 +544,7 @@ export async function admit( try { for (;;) { signal.throwIfAborted(); - if (await turn()) return; + if (await turn()) return id; } } finally { // A cancelled caller leaves the queue at once, rather than once its gate ends. @@ -526,6 +561,41 @@ export async function admit( } } +/** + * Until `signal` aborts, sample the CPU use of the admitted command in systemd scope `unit` every + * SETTLE_MS, and keep its admission's `cores` at the most CPUs it kept busy over one sample in the + * last RECENT_MS, rounded, so other callers count it at its busiest rather than at a lull. + */ +export async function measure(host: Host, admission: { id: string; unit: string }, signal: AbortSignal) { + const samples: { at: number; cores: number }[] = []; + let previous: { at: number; usage: number } | undefined; + let recorded = 0; + for (;;) { + try { + await host.sleep(SETTLE_MS, signal); + } catch (error) { + if (signal.aborted) return; + throw error; + } + const usage = host.usage(admission.unit); + const at = host.now(); + if (usage !== undefined) { + // A clock stepped backward yields no rate for its interval. + if (previous && at > previous.at) + samples.push({ at, cores: (usage - previous.usage) / ((at - previous.at) * 1000) }); + previous = { at, usage }; + } + while (samples.length > 0 && samples[0].at <= at - RECENT_MS) samples.shift(); + const cores = Math.round(Math.max(0, ...samples.map((sample) => sample.cores))); + for (let ledger = host.ledger(); cores !== recorded; ledger = host.ledger()) { + const mine = ledger.admissions.find(({ id }) => id === admission.id); + if (mine === undefined) throw new Error(`gate lost its admission ${admission.id} from the ledger`); + const updated = ledger.admissions.map((entry) => (entry === mine ? { ...mine, cores } : entry)); + if (host.publish(ledger, updated)) recorded = cores; + } + } +} + /** Marks an admitted command, so nested gated calls inside it run at once. */ const ADMITTED = "QUEST_GATE_ADMITTED"; @@ -587,20 +657,20 @@ export const inside: Owner = plain(true); * owner's kill stops the gate's scope too, and the gate forwards that owner's signals to it. */ const background = - (enclosing: string | null): Owner => + (unit: string, enclosing: string | null): Owner => (command, env, cancel) => - systemdScopes.run(`quest-gate-${randomUUID()}.scope`, process.cwd(), command, { + systemdScopes.run(unit, process.cwd(), command, { env, cancel, within: enclosing, slice: "background.slice", }); -/** Run the command in a background scope of its own, outside any Quest scope (see `background`). */ -export const scope: Owner = background(null); +/** Run the command in background scope `unit`, a new name, outside any Quest scope (see `background`). */ +export const scope = (unit: string): Owner => background(unit, null); -/** Run the command in a background scope bound to the Quest scope `enclosing` (see `background`). */ -export const bound = (enclosing: string): Owner => background(enclosing); +/** Run the command in background scope `unit`, a new name, bound to the Quest scope `enclosing` (see `background`). */ +export const bound = (unit: string, enclosing: string): Owner => background(unit, enclosing); /** * On Windows, run the command under a stub in a kill-on-close Job Object that forbids breakaway @@ -682,20 +752,22 @@ const TOKEN = 3; * The strongest owner this host offers. A nested call runs inside its admitted command, which an * enclosing Quest owner also signals when it is Quest's group or a Quest scope (see `inside`). A * call inside a Quest scope runs its command in a background scope bound to it. Linux owns descendants through a systemd scope, and settles for a process group, - * like macOS, without a systemd user manager. Windows owns them through a Job Object. + * like macOS, without a systemd user manager. Windows owns them through a Job Object. A systemd + * scope also lets the gate measure its command's CPU use (see `measure`), so `unit` names it. */ -export function owner(env = process.env, cgroup = "/proc/self/cgroup"): Owner { +export function owner(env = process.env, cgroup = "/proc/self/cgroup"): { run: Owner; unit?: string } { // A Quest scope around this process holds the command too, since it shares the cgroup or an ancestor. const enclosing = process.platform === "linux" ? questScope(cgroup) : undefined; - if (env[ADMITTED] === "1") return env[GROUP] === "1" || enclosing ? inside : direct; - if (process.platform === "win32") return job; - if (enclosing) return bound(enclosing.unit); - if (process.platform !== "linux") return group; + if (env[ADMITTED] === "1") return { run: env[GROUP] === "1" || enclosing ? inside : direct }; + if (process.platform === "win32") return { run: job }; + const unit = `quest-gate-${randomUUID()}.scope`; + if (enclosing) return { run: bound(unit, enclosing.unit), unit }; + if (process.platform !== "linux") return { run: group }; try { systemdScopes.check(); - return scope; + return { run: scope(unit), unit }; } catch { - return group; + return { run: group }; } } @@ -709,7 +781,8 @@ const notices: Record = { export async function gate(command: string[], options: { quick?: boolean } = {}): Promise { const cancel = interrupts(); try { - const run = owner(); + const { run, unit } = owner(); + let admitted: { host: Host; id: string } | undefined; // A nested invocation executes inside the already admitted command. if (process.env[ADMITTED] !== "1") { const shown = new Set(); @@ -723,13 +796,30 @@ export async function gate(command: string[], options: { quick?: boolean } = {}) priority: process.env[PRIORITY] === "1", quick: options.quick, }; - await admit(hostLimits(), fileHost(claimDirectory()), cancel.signal, notice, precedence); + const host = fileHost(claimDirectory()); + admitted = { host, id: await admit(hostLimits(), host, cancel.signal, notice, precedence) }; } catch (error) { if (cancel.signal.aborted) return cancelled(cancel.signal); throw error; } } - const result = await run(command, { ...process.env, [ADMITTED]: "1" }, cancel); + const stop = new AbortController(); + // Held as a value until the command ends, so a failure never goes unhandled meanwhile. + const measuring = + admitted !== undefined && unit !== undefined + ? measure(admitted.host, { id: admitted.id, unit }, stop.signal).then( + () => undefined, + (error: unknown) => ({ error }), + ) + : undefined; + let result: number; + try { + result = await run(command, { ...process.env, [ADMITTED]: "1" }, cancel); + } finally { + stop.abort(); + } + const failed = await measuring; + if (failed) throw failed.error; return cancel.signal.aborted ? cancelled(cancel.signal) : result; } finally { cancel.close(); diff --git a/src/cli/main.ts b/src/cli/main.ts index d823141e..2f7be586 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -129,7 +129,9 @@ export function main(argv: string[], out: Output): number | Promise { "Wait while the 1-minute load average exceeds the host's CPU count or available memory is below " + "min(4 GiB, RAM/4). Each admission in the last minute counts as one more unit of load and one " + "budget less memory: commands start at once while those still fit, and 5 seconds apart while " + - "only the raw measures do. The load limit applies only while a gated command still runs. Nested " + + "only the raw measures do. The load limit applies only while a gated command still runs. Where a " + + "systemd scope owns a running command, the load counts as at least the most CPUs it kept busy in " + + "the last minute, so a lull between its phases admits nothing more. Nested " + "calls run immediately. Limits come from the host; there is " + "no configuration.\n\n" + `--quick waits at most ${QUICK_MS / 1000} seconds, then starts the command anyway; pass it only for ` + diff --git a/src/cli/supervise.ts b/src/cli/supervise.ts index 98c33423..ff8ed50f 100644 --- a/src/cli/supervise.ts +++ b/src/cli/supervise.ts @@ -207,7 +207,7 @@ function signalScope(unit: string, signal: NodeJS.Signals, done: (error?: string } /** The scope's cgroup directory, or undefined once systemd no longer knows the unit. */ -function scopeGroup(unit: string): string | undefined { +export function scopeGroup(unit: string): string | undefined { const group = execFileSync("systemctl", ["--user", "show", "--property=ControlGroup", "--value", unit], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"], diff --git a/test/gate.test.ts b/test/gate.test.ts index bc3b25ed..ab01d79f 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -1,4 +1,5 @@ import { spawn, spawnSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; import { once } from "node:events"; import { chmodSync, @@ -30,6 +31,7 @@ import { type Host, hostLimits, linuxMemory, + measure, type Owner, owner, processStart, @@ -210,6 +212,65 @@ test("a running admission keeps its slot past the minute, and a finished one exp expect(host.ledger()).toEqual({ version: 100, admissions: [] }); }); +test("a running command that kept every CPU busy holds the host through a lull in the load", async () => { + const events: string[] = []; + // Between a build's phases, the load average falls below the host's CPUs. + const state = { now: 0, load: 1, memory: Infinity }; + const host = testHost(tempDir(), state, events); + const build = { id: "build", at: -10 * RECENT_MS, ...me, cores: 4 }; + expect(host.publish(host.ledger(), [build])).toBe(true); + const sleep = host.sleep; + host.sleep = async (ms, signal) => { + await sleep(ms, signal); + // Its busiest minute now left one CPU free. + const ledger = host.ledger(); + host.publish( + ledger, + ledger.admissions.map((admission) => (admission.id === build.id ? { ...build, cores: 3 } : admission)), + ); + }; + await admit({ load: 4, memory: GiB }, host, signal, (reason) => events.push(reason)); + expect(events).toEqual(["load", `sleep ${SETTLE_MS}`]); + expect(host.ledger().admissions).toEqual([ + { ...build, cores: 3 }, + { id: expect.any(String), at: SETTLE_MS, ...me }, + ]); +}); + +test("a gate records the most CPUs its command kept busy in the last minute", async () => { + const state = { now: 0, load: 0, memory: Infinity }; + const host = testHost(tempDir(), state); + host.publish(host.ledger(), [{ id: "build", at: 0, ...me }]); + // CPUs busy over each SETTLE_MS: before systemd creates the scope, the first reading, a + // build on four CPUs, then its tests on one for over a minute. + const busy = [undefined, 0, 4, 4, ...Array(RECENT_MS / SETTLE_MS).fill(1)]; + let usage: number | undefined; + let total = 0; + host.usage = () => usage; + const published: string[] = []; + const publish = host.publish; + host.publish = (base, admissions) => { + published.push(`${admissions[0].cores} at ${state.now}`); + return publish(base, admissions); + }; + const stop = new AbortController(); + host.sleep = async (ms) => { + if (busy.length === 0) { + stop.abort(); + throw stop.signal.reason; + } + state.now += ms; + const cores = busy.shift(); + if (cores === undefined) return; + total += cores * ms * 1000; + usage = total; + }; + await measure(host, { id: "build", unit: "quest-gate-build.scope" }, stop.signal); + // The build's last sample, at 20 s, holds its four CPUs for a minute. + expect(published).toEqual([`4 at ${3 * SETTLE_MS}`, `1 at ${4 * SETTLE_MS + RECENT_MS}`]); + expect(host.ledger().admissions).toEqual([{ id: "build", at: 0, ...me, cores: 1 }]); +}); + test("a priority caller waits for memory, never for load, even while a gated command runs", async () => { const directory = tempDir(); const state = { now: 0, load: 99, memory: 0 }; @@ -733,15 +794,17 @@ test("a nested call, or one inside a Quest scope, leaves its command to the encl }); expect(questScope(terminal)).toBeUndefined(); expect(questScope(cgroup(`${unit}-not`))).toBeUndefined(); - if (process.platform === "win32") return expect(owner({ QUEST_GATE_ADMITTED: "1" })).toBe(direct); + if (process.platform === "win32") return expect(owner({ QUEST_GATE_ADMITTED: "1" }).run).toBe(direct); // The enclosing owner signals the command too unless it was not one of Quest's. - expect(owner({ QUEST_GATE_ADMITTED: "1" }, terminal)).toBe(direct); - expect(owner({ QUEST_GATE_ADMITTED: "1", QUEST_GATE_GROUP: "1" }, terminal)).toBe(inside); + expect(owner({ QUEST_GATE_ADMITTED: "1" }, terminal)).toEqual({ run: direct }); + expect(owner({ QUEST_GATE_ADMITTED: "1", QUEST_GATE_GROUP: "1" }, terminal)).toEqual({ run: inside }); if (process.platform !== "linux") return; - expect(owner({ QUEST_GATE_ADMITTED: "1" }, gated)).toBe(inside); - expect(owner({ QUEST_GATE_ADMITTED: "1" }, child)).toBe(inside); - // An unadmitted call runs beside the enclosing scope, not inside it. - expect([direct, inside, scope]).not.toContain(owner({}, worktree)); + expect(owner({ QUEST_GATE_ADMITTED: "1" }, gated)).toEqual({ run: inside }); + expect(owner({ QUEST_GATE_ADMITTED: "1" }, child)).toEqual({ run: inside }); + // An unadmitted call runs beside the enclosing scope, not inside it, in a scope the gate names. + const beside = owner({}, worktree); + expect([direct, inside]).not.toContain(beside.run); + expect(beside.unit).toMatch(/^quest-gate-[0-9a-f-]+\.scope$/); }); /** Gone, or a zombie its new parent has not reaped yet: either way it runs no code. */ @@ -770,10 +833,13 @@ function descendant(dir: string, rest: string, { leave = false } = {}) { }; } +/** A systemd scope owner with a new unit, as the gate names one. */ +const scoped: Owner = (command, env, cancel) => scope(`quest-gate-${randomUUID()}.scope`)(command, env, cancel); + const owners: [string, Owner][] = process.platform === "linux" ? [ - ["a systemd scope", scope], + ["a systemd scope", scoped], ["a process group", group], ] : process.platform === "win32" @@ -1016,9 +1082,22 @@ describe.skipIf(process.platform === "win32")("an outermost gate owning a proces describe.runIf(process.platform === "linux")("a systemd scope", () => { test("also stops a descendant that left its process group", { timeout: 30_000 }, async () => { const { command, pid } = descendant(tempDir(), "exit 3", { leave: true }); - expect(await scope(command, process.env, cancellation().cancel)).toBe(3); + expect(await scoped(command, process.env, cancellation().cancel)).toBe(3); await until(() => stopped(pid()), "the daemon to stop"); }); + + test("reports its CPU use to the gate until it ends", { timeout: 30_000 }, async () => { + const host = fileHost(tempDir()); + const unit = `quest-gate-${randomUUID()}.scope`; + const done = join(tempDir(), "done"); + expect(host.usage(unit)).toBeUndefined(); + const spin = ["/bin/sh", "-c", `while [ ! -e ${quote(done)} ]; do :; done`]; + const result = scope(unit)(spin, process.env, cancellation().cancel); + await until(() => (host.usage(unit) ?? 0) > 0, "the scope's CPU use to show"); + writeFileSync(done, ""); + expect(await result).toBe(0); + await until(() => host.usage(unit) === undefined, "systemd to remove the scope"); + }); }); /** @@ -1035,20 +1114,21 @@ async function insideScope(launch: (command: string[], env: NodeJS.ProcessEnv) = script, `import { spawnSync } from "node:child_process"; import { existsSync, readFileSync, writeFileSync } from "node:fs"; - import { direct, group, inside, job, owner, scope } from ${JSON.stringify(gate)}; + import { direct, group, inside, job, owner } from ${JSON.stringify(gate)}; const dir = ${JSON.stringify(dir)}; - const own = owner(process.env); - const kind = new Map([[direct, "direct"], [group, "group"], [inside, "inside"], [job, "job"], [scope, "scope"]]).get(own) ?? "bound"; + const { run: own, unit } = owner(process.env); + const kind = unit ? "scope" : new Map([[direct, "direct"], [group, "group"], [inside, "inside"], [job, "job"]]).get(own); let report = { kind }; if (${run}) { const never = new AbortController().signal; - const command = "cat /proc/self/cgroup > " + dir + "/cgroup; sleep 1000 & echo $! > " + dir + "/pid; exit 7"; + const command = "cat /proc/self/cgroup > " + dir + "/cgroup; systemctl --user show --property=BindsTo --value " + unit + " > " + dir + "/binds; sleep 1000 & echo $! > " + dir + "/pid; exit 7"; const status = await own(["/bin/sh", "-c", command], process.env, { signal: never, kill: never }); const path = readFileSync(dir + "/cgroup", "utf8").trim().slice(3); const pid = readFileSync(dir + "/pid", "utf8").trim(); const ps = spawnSync("ps", ["-o", "stat=", "-p", pid], { encoding: "utf8" }); const stopped = ps.status !== 0 || ps.stdout.trim().startsWith("Z"); - report = { kind, status, path, stopped, removed: !existsSync("/sys/fs/cgroup" + path) }; + const binds = readFileSync(dir + "/binds", "utf8").trim(); + report = { kind, status, path, binds, stopped, removed: !existsSync("/sys/fs/cgroup" + path) }; } writeFileSync(${JSON.stringify(result)}, JSON.stringify(report));`, ); @@ -1060,8 +1140,10 @@ async function insideScope(launch: (command: string[], env: NodeJS.ProcessEnv) = describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () => { test("runs its command in a background scope beside the enclosing one", { timeout: 30_000 }, async () => { - const report = await insideScope((command, env) => scope(command, env, cancellation().cancel), true); - expect(report).toMatchObject({ kind: "bound", status: 7, stopped: true, removed: true }); + const report = await insideScope((command, env) => scoped(command, env, cancellation().cancel), true); + expect(report).toMatchObject({ kind: "scope", status: 7, stopped: true, removed: true }); + // Bound to the enclosing scope, a gate's scope here, so that scope's owner stops it too. + expect(report.binds).toMatch(/^quest-gate-[0-9a-f-]+\.scope$/); // A sibling, so only the gated command yields the CPU, not the rest of the enclosing scope. expect(report.path).toMatch(/\/background\.slice\/quest-gate-[0-9a-f-]+\.scope$/); }); @@ -1077,7 +1159,7 @@ describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () = `trap '' TERM; sleep 1000 & echo $! > ${quote(pid)}; : > ${quote(ready)}; wait`, ]; const { stop, kill, cancel } = cancellation(); - const result = scope(nestedGate(dir, command), outside(), cancel); + const result = scoped(nestedGate(dir, command), outside(), cancel); await until(() => existsSync(ready), "the command to start"); stop.abort("SIGTERM"); kill.abort(); @@ -1100,7 +1182,7 @@ describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () = `import { existsSync } from "node:fs"; import { direct, inside, owner } from ${JSON.stringify(gate)}; import { cancelled, interrupts } from ${JSON.stringify(interrupt)}; - const own = owner(process.env); + const { run: own } = owner(process.env); if (own === direct || own === inside) process.exit(90); const signals = interrupts(); const stop = new AbortController(); @@ -1130,7 +1212,7 @@ describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () = const dir = tempDir(); const leaf = signalled(dir, ["SIGTERM"], 200); const { stop, cancel } = cancellation(); - const result = scope(nestedGate(dir, phase(dir, leaf.script)), outside(), cancel); + const result = scoped(nestedGate(dir, phase(dir, leaf.script)), outside(), cancel); await created(leaf.ready); stop.abort("SIGTERM"); expect(await result).toBe(128 + constants.signals.SIGTERM); @@ -1145,7 +1227,7 @@ describe.runIf(process.platform === "linux")("a gate inside a Quest scope", () = const abort = join(dir, "abort"); // The shell waits on the leaf in the background, so only a signal to the whole cgroup reaches it. const command = ["/bin/sh", "-c", `${quote(process.execPath)} ${quote(leaf.script)} & wait`]; - const result = scope(nestedGate(dir, command, abort), outside(), cancellation().cancel); + const result = scoped(nestedGate(dir, command, abort), outside(), cancellation().cancel); await created(leaf.ready); writeFileSync(abort, ""); expect(await result).toBe(128 + constants.signals.SIGTERM); @@ -1267,7 +1349,7 @@ test("the checkout's Bun config hands a gated command no .env files from its dir describe.runIf(process.platform === "win32")("a job object", () => { test("is the gate's owner outside an admitted command", () => { - expect(owner({})).toBe(job); + expect(owner({})).toEqual({ run: job }); }); test("keeps the exact exit status and stops a detached descendant", { timeout: 30_000 }, async () => { diff --git a/test/worktree.run.test.ts b/test/worktree.run.test.ts index bf4d7706..5b7589cd 100644 --- a/test/worktree.run.test.ts +++ b/test/worktree.run.test.ts @@ -1,6 +1,7 @@ // Supervised runs against the host's real systemd user manager. On Linux these need 5.14+, systemd 254+ // and unified cgroup v2; a host without them fails here rather than skipping. import { execFileSync, spawn, spawnSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; import { once } from "node:events"; import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; import { join, resolve } from "node:path"; @@ -173,7 +174,7 @@ describe.runIf(process.platform === "linux")("supervised runs under systemd", { writeFileSync(justfile, `run:\n ${recipe.map((word) => JSON.stringify(word)).join(" ")}\n`); const stop = new AbortController(); const kill = new AbortController(); - const result = scope( + const result = scope(`quest-gate-${randomUUID()}.scope`)( ["just", "--justfile", justfile, "--working-directory", dir, "run"], { ...process.env, QUEST_WORKTREE_HOME: home, QUEST_GATE_ADMITTED: "1" }, { signal: stop.signal, kill: kill.signal }, From 49bcb893e94a9c760b8115e48fd3a36b7f2d8c0a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 11:20:26 -0800 Subject: [PATCH 8/9] fix(gate): count a command's CPU burst before its first sample Co-Authored-By: Claude Opus 5.5 --- src/cli/gate.ts | 6 +++--- test/gate.test.ts | 20 +++++++++++++++----- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/src/cli/gate.ts b/src/cli/gate.ts index 812fc6de..abd76f24 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -568,7 +568,8 @@ export async function admit( */ export async function measure(host: Host, admission: { id: string; unit: string }, signal: AbortSignal) { const samples: { at: number; cores: number }[] = []; - let previous: { at: number; usage: number } | undefined; + // The command's scope starts with no CPU use, so a burst before the first sample still counts. + let previous = { at: host.now(), usage: 0 }; let recorded = 0; for (;;) { try { @@ -581,8 +582,7 @@ export async function measure(host: Host, admission: { id: string; unit: string const at = host.now(); if (usage !== undefined) { // A clock stepped backward yields no rate for its interval. - if (previous && at > previous.at) - samples.push({ at, cores: (usage - previous.usage) / ((at - previous.at) * 1000) }); + if (at > previous.at) samples.push({ at, cores: (usage - previous.usage) / ((at - previous.at) * 1000) }); previous = { at, usage }; } while (samples.length > 0 && samples[0].at <= at - RECENT_MS) samples.shift(); diff --git a/test/gate.test.ts b/test/gate.test.ts index ab01d79f..eb191d88 100644 --- a/test/gate.test.ts +++ b/test/gate.test.ts @@ -237,13 +237,11 @@ test("a running command that kept every CPU busy holds the host through a lull i ]); }); -test("a gate records the most CPUs its command kept busy in the last minute", async () => { +/** Run `measure` over `busy`, the CPUs its command keeps busy over each SETTLE_MS, undefined before its scope exists; returns what it publishes. */ +async function measured(busy: (number | undefined)[]) { const state = { now: 0, load: 0, memory: Infinity }; const host = testHost(tempDir(), state); host.publish(host.ledger(), [{ id: "build", at: 0, ...me }]); - // CPUs busy over each SETTLE_MS: before systemd creates the scope, the first reading, a - // build on four CPUs, then its tests on one for over a minute. - const busy = [undefined, 0, 4, 4, ...Array(RECENT_MS / SETTLE_MS).fill(1)]; let usage: number | undefined; let total = 0; host.usage = () => usage; @@ -266,9 +264,21 @@ test("a gate records the most CPUs its command kept busy in the last minute", as usage = total; }; await measure(host, { id: "build", unit: "quest-gate-build.scope" }, stop.signal); + return { published, admissions: host.ledger().admissions }; +} + +test("a gate records the most CPUs its command kept busy in the last minute", async () => { + // Before systemd creates the scope, an idle first reading, a build on four CPUs, then its + // tests on one for over a minute. + const { published, admissions } = await measured([undefined, 0, 4, 4, ...Array(RECENT_MS / SETTLE_MS).fill(1)]); // The build's last sample, at 20 s, holds its four CPUs for a minute. expect(published).toEqual([`4 at ${3 * SETTLE_MS}`, `1 at ${4 * SETTLE_MS + RECENT_MS}`]); - expect(host.ledger().admissions).toEqual([{ id: "build", at: 0, ...me, cores: 1 }]); + expect(admissions).toEqual([{ id: "build", at: 0, ...me, cores: 1 }]); +}); + +test("a gate records its command's burst before the first sample", async () => { + const { published } = await measured([4, ...Array(RECENT_MS / SETTLE_MS).fill(0)]); + expect(published).toEqual([`4 at ${SETTLE_MS}`, `0 at ${SETTLE_MS + RECENT_MS}`]); }); test("a priority caller waits for memory, never for load, even while a gated command runs", async () => { From f356b6e18d26c63a1db59caa8346346d9d3ff2bc Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 10 Nov 2026 11:47:02 -0800 Subject: [PATCH 9/9] fix(gate): keep a cancelled command's status over a measurement failure; fix help wording Co-Authored-By: Claude Opus 5.5 --- src/cli/gate.ts | 9 +++++++-- src/cli/main.ts | 5 +++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/src/cli/gate.ts b/src/cli/gate.ts index abd76f24..5395d373 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -777,7 +777,11 @@ const notices: Record = { queue: "quest gate: waiting for commands queued earlier to start", }; -/** Run from the CLI. */ +/** + * Run from the CLI. A failure to measure the command is fatal even when the command succeeded, + * since every other gate on the host relies on that accounting; a cancelled command still + * reports its cancellation. + */ export async function gate(command: string[], options: { quick?: boolean } = {}): Promise { const cancel = interrupts(); try { @@ -819,8 +823,9 @@ export async function gate(command: string[], options: { quick?: boolean } = {}) stop.abort(); } const failed = await measuring; + if (cancel.signal.aborted) return cancelled(cancel.signal); if (failed) throw failed.error; - return cancel.signal.aborted ? cancelled(cancel.signal) : result; + return result; } finally { cancel.close(); } diff --git a/src/cli/main.ts b/src/cli/main.ts index edd2c9a9..ad412698 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -130,8 +130,9 @@ export function main(argv: string[], out: Output): number | Promise { "min(4 GiB, RAM/4). Each admission in the last minute counts as one more unit of load and one " + "budget less memory: commands start at once while those still fit, and 5 seconds apart while " + "only the raw measures do. The load limit applies only while a gated command still runs. Where a " + - "systemd scope owns a running command, the load counts as at least the most CPUs it kept busy in " + - "the last minute, so a lull between its phases admits nothing more. Nested " + + "systemd scope owns each running command, the load counts as at least the sum of their peak CPU " + + "use in the last minute, and nothing starts while those fill every CPU, so a lull between a " + + "command's phases does not free its CPUs. Nested " + "calls run immediately. Limits come from the host; there is " + "no configuration.\n\n" + `--quick waits at most ${QUICK_MS / 1000} seconds, then starts the command anyway; pass it only for ` +