From 8eda31e50e790bf709d48f3dd631a43b262a155c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 14:06:11 -0700 Subject: [PATCH 1/2] chore(quest): claim quest/m0/local-gate.md Quest-Claim: 470bfc40d20c751d From 951b698d93aeafa85826642e8327b9e9b263cc1a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 21 Oct 2026 14:31:24 -0700 Subject: [PATCH 2/2] test(gate): measure admission under Worker suite contention Add scripts/gate-bench.sh, which starts N Worker suites at once in detached worktrees, gated or bypassing the gate, optionally pinned to a few CPUs, and reports failed suites, 5-second timeouts, waits, run times, makespan, and peak load, CPU pressure, and least available memory. It also measures admission overhead against a bypassed call. The measurements finish the gate quest, so delete it and its links. Correct the README: waiters are admitted about 10 seconds apart, since a claim is refused while the previous window's claim exists. Co-Authored-By: Claude Opus 5.5 --- README.md | 5 +- quest/m0/README.md | 1 - quest/m0/helpers/README.md | 1 - quest/m0/helpers/headless-agents.md | 1 - quest/m0/local-gate.md | 59 ------------- quest/m0/worktree-agents.md | 1 - quest/m0/worktree.md | 1 - scripts/gate-bench.sh | 124 ++++++++++++++++++++++++++++ 8 files changed, 127 insertions(+), 66 deletions(-) delete mode 100644 quest/m0/local-gate.md create mode 100755 scripts/gate-bench.sh diff --git a/README.md b/README.md index 6be86111..5ec9b88d 100644 --- a/README.md +++ b/README.md @@ -381,8 +381,9 @@ smaller hosts, where RAM is capped by any container memory limit. On Linux, available memory includes reclaimable host and cgroup file cache and is bounded by the remaining budget of each enclosing memory cgroup. Load and memory count work from every session and user. The load average lags new work, so the gate -admits at most one waiting command every 5 seconds per host rather than -releasing every waiter at once. Windows reports no load +admits at most one waiting command per 5-second window per host, never in +consecutive windows, rather than releasing every waiter at once: a burst of +waiters starts about 10 seconds apart. Windows reports no load average and macOS counts reclaimable cache as used memory, so those signals are skipped there. diff --git a/quest/m0/README.md b/quest/m0/README.md index 589735b7..6bb6e993 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -35,7 +35,6 @@ ownership-manifest design: ## Required -- [Gate contention measurements](/quest/m0/local-gate.md) - evidence that gated checks stay responsive under shared load - [Worktree process ownership](/quest/m0/worktree.md) - supervise checkout commands and release checkouts automatically - [Agent lifecycle integration](/quest/m0/worktree-agents.md) - reclaim completed and suspended checkouts through their harness - [Agent workflow helpers](/quest/m0/helpers/README.md) - CLI commands for CI waits, PR status, claims, quest moves, and serialized merges diff --git a/quest/m0/helpers/README.md b/quest/m0/helpers/README.md index 9bf1e552..cb8b269d 100644 --- a/quest/m0/helpers/README.md +++ b/quest/m0/helpers/README.md @@ -53,5 +53,4 @@ commands from Sep-Oct. Session counts in each child come from that window. ## Related -- [Expensive command gate](/quest/m0/local-gate.md) - CPU admission for the checks these helpers wrap - [Agent lifecycle integration](/quest/m0/worktree-agents.md) - removes the agent-level load-average rules diff --git a/quest/m0/helpers/headless-agents.md b/quest/m0/helpers/headless-agents.md index ff4e2b04..f1b6d906 100644 --- a/quest/m0/helpers/headless-agents.md +++ b/quest/m0/helpers/headless-agents.md @@ -63,5 +63,4 @@ Recommended shape, open for review: ## Related - [Agent lifecycle integration](/quest/m0/worktree-agents.md) - removes the skill's load-average launch rules, so neither the skill nor the command polls load -- [Expensive command gate](/quest/m0/local-gate.md) - admits expensive phases inside each agent, so the command needs no load polling - [Runner](/quest/m0/kix/runner.md) - the kix runner's headless Claude Code path; share the adapter diff --git a/quest/m0/local-gate.md b/quest/m0/local-gate.md deleted file mode 100644 index 998c5fd1..00000000 --- a/quest/m0/local-gate.md +++ /dev/null @@ -1,59 +0,0 @@ -# [S] Measure quest gate under representative contention - -## Goal - -`quest gate ` holds expensive commands for one user on one host until the -host has CPU and memory room, shared across repositories and worktrees. -Concurrent agent sessions remain useful without multiplying full-machine builds -and test stacks. - -Load and memory admission and its automatic use by `just check` / `just test` -have landed. What remains is measuring representative build/test contention -with and without the gate, for example how many concurrent Worker suites hit -their 5-second timeouts, before claiming a responsiveness improvement. - -## Plan - -- Decided 2026-10-15: gate expensive command phases, not agent lifetimes. - Repository entrypoints invoke the gate automatically; ordinary reasoning and - I/O are not gated. No adaptive scheduler or new daemon is needed for this scope. -- Decided 2026-10-15 (user): gate checks and CI-like suites on host load, not - agents, because load from every session and user is what starves test timers. - The agent-level load-average rule was removed from the spawn and complete skills. -- Decided 2026-10-21 (user): admission is load and memory only, on every - platform. This supersedes the 2026-10-15 decisions to add Linux systemd scope - slots on top: every review round on the scope handshake, `cgroup.kill` inode - pinning, and manager probes found another race, for little admission benefit. - Exact descendant ownership and orphan cleanup move to the - [worktree process-ownership quest](/quest/m0/worktree.md). The removed - backend, its probes, and its opt-in systemd suite are recoverable from commit - 5684307 in [kixelated/quest#128](https://github.com/kixelated/quest/pull/128). -- Decided 2026-10-15 (user): Bun's `--no-orphans` stays a research option. Its - best-effort registration and bounded PID-tree walks do not establish exact - ownership by themselves. -- Decided 2026-10-15: at most one admission per 5-second window across all - callers, claimed by exclusively creating a per-window file. A claim is - refused while the previous window's claim exists, so admissions are always a - full window apart (revised 2026-10-21 after a Codex finding). The load average - lags new work and is sampled every 5 seconds, so waiters that all see it fall - must not start together. Nothing is held across the command, so a crash - leaves no stale lock. Claims live in `$XDG_RUNTIME_DIR`, or a per-hostname - cache directory, because a home directory may be shared between hosts. -- Decided 2026-10-15 (user): limits come from host specs, with no configuration - file. The load limit is the host's CPU count, because the load average counts - every CPU, including ones outside this process's affinity (revised - 2026-10-21 from the usable core count). Admission waits while available - memory is below `min(4 GiB, RAM / 4)`: 4 GiB is a typical build or test - stack, and a small host still runs one command. RAM is capped by a cgroup - memory limit, and cgroup headroom subtracts inactive file cache from usage, - since it is reclaimable. macOS counts reclaimable cache as used memory, so - only its load is gated; Windows reports no load average, so only its memory is. -- Nested gated calls run immediately, identified by a marker the gate sets in - the admitted command's environment. Fairness is not promised. -- Measure admission overhead alongside contention; the earlier 37 ms figure - included the removed systemd probe. - -## Related - -- [Worktree process ownership](/quest/m0/worktree.md) - owns checkout command descendants, with its own process-ownership backend -- [Own-PC runner](/quest/m0/kix/runner.md) - can gate expensive job phases on host room diff --git a/quest/m0/worktree-agents.md b/quest/m0/worktree-agents.md index 804ff316..88d3b086 100644 --- a/quest/m0/worktree-agents.md +++ b/quest/m0/worktree-agents.md @@ -40,7 +40,6 @@ merge. ## Required - [Worktree process ownership](/quest/m0/worktree.md) - supervised commands and automatic scratch release -- [Expensive command gate](/quest/m0/local-gate.md) - resource control before removing agent-count throttles ## Related diff --git a/quest/m0/worktree.md b/quest/m0/worktree.md index dd0c07a2..c90974aa 100644 --- a/quest/m0/worktree.md +++ b/quest/m0/worktree.md @@ -48,5 +48,4 @@ work, stealing a live lock, or deleting scratch a running process still uses. ## Related -- [Gate contention measurements](/quest/m0/local-gate.md) - independent CPU and memory admission - [Agent lifecycle integration](/quest/m0/worktree-agents.md) - native agents and workflow adoption diff --git a/scripts/gate-bench.sh b/scripts/gate-bench.sh new file mode 100755 index 00000000..1d081dbc --- /dev/null +++ b/scripts/gate-bench.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash +# Measure `quest gate` under contention. Run inside `nix develop`. +# +# scripts/gate-bench.sh overhead [runs] +# Admission cost: `quest gate true` with a fresh claim directory, so no +# window is ever taken, against the same call with the gate bypassed. +# scripts/gate-bench.sh gated|ungated [cpus] +# Start Worker test suites at once, each in its own detached +# worktree of HEAD, through the gate or bypassing it. `cpus` is a taskset +# list that confines the suites to a few cores, to oversubscribe them +# without saturating a shared host; the gate still sees host-wide load. +# +# Reports failed suites, 5-second test timeouts, admission waits, run times, +# the makespan, and the peak 1-minute load and CPU pressure and least available +# memory while the suites ran. Other work on the host counts too, so record the +# load printed first. +set -euo pipefail + +root=$(git -C "$(dirname -- "${BASH_SOURCE[0]}")" rev-parse --show-toplevel) +unset QUEST_GATE_ADMITTED +work=$(mktemp -d "${TMPDIR:-/tmp}/quest-gate-bench.XXXXXX") +sampler= +cleanup() { + [[ -z $sampler ]] || kill "$sampler" 2>/dev/null || true + for tree in "$work"/tree.*; do + [[ ! -d $tree ]] || git -C "$root" worktree remove --force "$tree" + done + rm -rf "$work" +} +trap cleanup EXIT + +usage() { + echo "usage: $0 overhead [runs] | $0 gated|ungated [cpus]" >&2 + exit 2 +} + +now_ms() { echo $((${EPOCHREALTIME/./} / 1000)); } + +# Print `_median=` and `_max=` of the remaining arguments. +stats() { + local name=$1 + shift + printf '%s\n' "$@" | sort -n | awk -v n="$name" \ + '{ v[NR] = $1 } END { printf "%s_median=%d %s_max=%d ", n, v[int((NR + 1) / 2)], n, v[NR] }' +} + +overhead() { + local runs=${1:-20} i start bypass=() gated=() + cd "$root" + for ((i = 0; i < runs; i++)); do + start=$(now_ms) + QUEST_GATE_ADMITTED=1 bun run --silent quest gate true + bypass+=($(($(now_ms) - start))) + start=$(now_ms) + XDG_RUNTIME_DIR="$work/claims.$i" bun run --silent quest gate true + gated+=($(($(now_ms) - start))) + done + echo "runs=$runs $(stats bypass_ms "${bypass[@]}")$(stats gated_ms "${gated[@]}")" +} + +contend() { + local mode=$1 suites=${2:-} cpus=${3:-} i start pin=() pids=() + [[ $suites =~ ^[1-9][0-9]*$ ]] || usage + [[ -z $cpus ]] || pin=(taskset -c "$cpus") + + for ((i = 0; i < suites; i++)); do + git -C "$root" worktree add --quiet --detach "$work/tree.$i" HEAD + (cd "$work/tree.$i" && bun install --frozen-lockfile >/dev/null) + done + + echo "mode=$mode suites=$suites cpus=${cpus:-all} host_cpus=$(nproc --all) load_before=$(cut -d' ' -f1-3 /proc/loadavg)" + start=$(now_ms) + ( + while :; do + printf '%s %s %s\n' "$(cut -d' ' -f1 /proc/loadavg)" \ + "$(sed -n 's/^some avg10=\([0-9.]*\).*/\1/p' /proc/pressure/cpu 2>/dev/null || echo 0)" \ + "$(awk '/^MemAvailable:/ { print $2 }' /proc/meminfo)" + sleep 1 + done + ) >"$work/samples" & + sampler=$! + + for ((i = 0; i < suites; i++)); do + ( + cd "$work/tree.$i" + begin=$(now_ms) + [[ $mode == gated ]] || export QUEST_GATE_ADMITTED=1 + status=0 + # shellcheck disable=SC2016 # The admitted command expands it. + "${pin[@]}" timeout 900 bun run --silent quest gate \ + bash -c 'echo $((${EPOCHREALTIME/./} / 1000)) >admitted; exec bun run --cwd cloudflare test' \ + >log 2>&1 || status=$? + echo "$begin $(cat admitted 2>/dev/null || echo "$begin") $(now_ms) $status" >result + ) & + pids+=($!) + done + wait "${pids[@]}" + kill "$sampler" + wait "$sampler" || true + sampler= + + local failed=0 timeouts=0 waits=() runs=() last=0 begin admitted end status + for ((i = 0; i < suites; i++)); do + read -r begin admitted end status <"$work/tree.$i/result" + ((status == 0)) || failed=$((failed + 1)) + timeouts=$((timeouts + $(grep -c 'timed out in [0-9]*ms' "$work/tree.$i/log" || true))) + waits+=($((admitted - begin))) + runs+=($((end - admitted))) + ((end < last)) || last=$end + done + printf 'failed_suites=%d/%d test_timeouts=%d %s%s' "$failed" "$suites" "$timeouts" \ + "$(stats wait_ms "${waits[@]}")" "$(stats run_ms "${runs[@]}")" + awk -v span=$((last - start)) '$1 > l { l = $1 } $2 > p { p = $2 } NR == 1 || $3 < m { m = $3 } + END { printf "makespan_ms=%d peak_load1=%s peak_cpu_psi10=%s min_mem_available_gib=%.1f\n", span, l, p, m / 2^20 }' \ + "$work/samples" + # The tests that failed most often, with how many suites they failed in. + grep -h '^ *FAIL ' "$work"/tree.*/log | sed 's/^ *//' | sort | uniq -c | sort -rn | head -n 10 || true +} + +case ${1:-} in + overhead) overhead "${@:2}" ;; + gated | ungated) contend "$@" ;; + *) usage ;; +esac