diff --git a/quest/m0/README.md b/quest/m0/README.md index 23b57535..8387cfbe 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -35,6 +35,7 @@ ownership-manifest design: ## Required +- [Gate headroom](/quest/m0/gate-headroom.md) - gated commands start at once while the host has room - [Worktree removal](/quest/m0/worktree.md) - remove finished checkouts without losing work - [Agent lifecycle integration](/quest/m0/worktree-agents.md) - reclaim completed and suspended checkouts through their harness - [Agent workflow helpers](/quest/m0/helpers/README.md) - CLI commands for CI waits, PR status, claims, quest moves, and serialized merges diff --git a/quest/m0/gate-headroom.md b/quest/m0/gate-headroom.md new file mode 100644 index 00000000..6dfe24a4 --- /dev/null +++ b/quest/m0/gate-headroom.md @@ -0,0 +1,99 @@ +# [M] Admit gated commands at once while the host has room + +## Goal + +`quest gate` adds no noticeable delay while the host has room, and still +spaces admissions out as the host nears its CPU or memory limit. A burst of +gated `just test` runs on an idle host starts together, instead of about +10 seconds apart. + +## Plan + +Decided 2026-10-21 (user) after the contention measurements in +[kixelated/quest#177](https://github.com/kixelated/quest/pull/177), reproducible +with `scripts/gate-bench.sh`. On a 24-core host, Worker suites only timed out +at about 8 suites per CPU, where the gate prevented the failures at about +1.3× the ungated makespan. With room to spare, the gate only added waiting: a +burst of 8 took 104 s gated versus 19 s ungated. + +- **Admit with headroom (user).** The load average and free memory both lag + new work, so count each admission from the last minute as one more unit of + load and as one command's memory budget taken from available memory. Then: + 1. Admit at once while load plus recent admissions stays under the CPU + count, and available memory minus the recent admissions' budgets still + covers one command. + 2. Once recent admissions tip either measure over, but the load alone is at + or under the CPU count and available memory alone covers one command, + admit one command at a time, `SETTLE_MS` apart. Spaced admissions count + as recent too. + 3. Otherwise wait, as today. + + This reverses the 2026-10-15 "no adaptive scheduler" decision. Memory + admission stays, because heavier repositories (Rust builds) are what risk + OOM; counting recent admissions against it keeps a burst from passing on + one shared free-memory sample. Rejected: fixing the spacing only, taking + the gate out of `just check` and `just test`, and keeping today's delay. +- **Admission requirements.** Today a claim is one `O_EXCL` file per 5-second + window, and each admission deletes the older ones; the replacement for + `Host.claim(window)` is left to the implementation, which must meet these: + - Count and claim atomically across processes, with several admissions per + window and each admission remembered for a minute. + - Never admit past the three-band rule, and always admit some waiter when + the rule has room. + - Recover from a caller that crashes mid-admission on every platform the + gate runs on, without blocking admission indefinitely. + - Bound the effect of a wall-clock step in either direction to about one + record lifetime. + - Space band-2 admissions at least `SETTLE_MS` apart, measured from the + previous admission's time (decision 8 in + [kixelated/quest#128](https://github.com/kixelated/quest/pull/128)), with + waiters waking when that interval ends rather than at the next window + boundary, so queued commands stop starting about 10 s apart. + - Update the README's "Gate expensive commands" admission paragraph and the + `admit()` doc comment, which both describe today's behaviour. +- **Hazards to test.** Each came up in review of the plan in + [kixelated/quest#180](https://github.com/kixelated/quest/pull/180): + - A burst passing one shared free-memory sample, so every caller fits a + budget that only one does. + - A burst that all reads "0 recent" before any of them records. + - Two admissions milliseconds apart across a window boundary. + - Expired records leaving holes in numbered slots, so a slot number + undercounts recent admissions. + - Every caller in an over-capacity burst recording, seeing no room, and + withdrawing, then repeating that together forever. + - A dead lock holder's PID reused on macOS or Windows, where only Linux + records a process start time, keeping the lock held indefinitely. + - A backward clock step leaving records dated far in the future, which + count as recent indefinitely or are discarded while still valid. + - A forward clock step past the record lifetime, expiring records early and + dropping their spacing and reservations. + - `performance.now()` differing per process, and `CLOCK_MONOTONIC` + restarting at boot while the fallback claim directory persists. +- **Stable claim directory.** Callers with and without `XDG_RUNTIME_DIR` use + different claim directories today. Every caller of one user on one host + must use the same claim location, one that never switches with the + caller's environment or with login sessions, and that stays per-host when + home directories are shared. `gate-bench.sh overhead` needs another way to + get a fresh directory. Hazards to test: callers with and without + `XDG_RUNTIME_DIR`, and `/run/user/` appearing or disappearing as logind + sessions start and end while background callers keep running. +- **Ctrl-C once.** In both `quest gate` and `quest worktree run` + (`src/cli/supervise.ts`), a terminal Ctrl-C reaches the child directly and + again through the forwarder. Share one fix between them. On Unix, while + Quest is in the terminal's foreground process group, skip forwarding only + SIGINT, which the terminal also delivers to the child; always forward SIGHUP + and SIGTERM, which may be sent to Quest alone. A SIGINT sent with `kill` to + Quest alone is then not forwarded. On Windows, + Ctrl-C reaches every process on the console and Node's `child.kill()` + terminates the child forcefully, so `quest gate` waits for the child + instead of forwarding; `quest worktree run` is Linux-only. +- Re-run `scripts/gate-bench.sh` for the idle-burst and 8× cases and record + both results in the PR. Expect the pinned 8× case to look ungated: it + confines 16 suites to two CPUs with `taskset`, while headroom is measured + across the host, so most start at once and the timeouts from + [kixelated/quest#177](https://github.com/kixelated/quest/pull/177) likely + return. That follows from the decision and is not a regression. + +## Related + +- [Agent lifecycle integration](/quest/m0/worktree-agents.md) - removes agent-count throttles once resource control is in place diff --git a/quest/m0/kix/inbox-accuracy.md b/quest/m0/kix/inbox-accuracy.md index f24377e3..345635ce 100644 --- a/quest/m0/kix/inbox-accuracy.md +++ b/quest/m0/kix/inbox-accuracy.md @@ -31,4 +31,8 @@ Decided 2026-10-21 from the follow-ups of loses closed history). - Extend the seeded dev server (`cloudflare/dev/seed.ts`) with a claimed draft, a draft that hands back decisions, and an `UNKNOWN` PR, so - `test/seed.test.ts` covers each state. + `cloudflare/test/seed.test.ts` covers each state. +- Keep `cloudflare/test/seed.test.ts` within the Worker suite's 5-second test + timeout as the seed grows; it was the only test to time out under 8× CPU + overload in [kixelated/quest#177](https://github.com/kixelated/quest/pull/177). + Slim it rather than raising its timeout. diff --git a/quest/m0/worktree.md b/quest/m0/worktree.md index bff617d4..bbe0857a 100644 --- a/quest/m0/worktree.md +++ b/quest/m0/worktree.md @@ -38,8 +38,7 @@ live and refuses a checkout that still holds anything not already preserved. lingering until the next successful release. - Out of scope: terminal Ctrl-C reaches the command twice (once from the terminal, once forwarded by the supervisor). It is shared with `quest gate` - and planned separately in - [kixelated/quest#180](https://github.com/kixelated/quest/pull/180). + and planned separately in [Gate headroom](/quest/m0/gate-headroom.md). - Measured 2026-10-21 on this repository: a fresh checkout is 2.9 MB, 394 MB after installing dependencies and building, and 2.9 MB once `run` released it. A suspended job keeps the same released footprint until it is restored.