diff --git a/README.md b/README.md index 5aa9eede..868d3c88 100644 --- a/README.md +++ b/README.md @@ -317,6 +317,48 @@ same facts as one object with the fields `url`, `state`, `draft`, `head`, as in `quest wait`. Any PR state exits 0, invalid arguments exit 2, and other failures exit 1. +## Prunable local checkouts + +The experimental `quest worktree` commands share one disk budget across this +user's repositories and release a checkout's disk by deleting its declared +scratch in place. They only operate on Git worktrees they create; use the +owning application for Codex, Claude, or other managed checkouts. Process +supervision remains unfinished: stop agents and commands in a checkout before +releasing it. + +Set a shared capacity and an explicit reservation in bytes, covering the +checkout and its expected build output. Measure representative jobs before +choosing the budget. Reservations serialize creators and account for concurrent +expected growth. They do not enforce filesystem quotas against arbitrary +writers, and Git objects stored in the shared repository are not charged. + +```sh +quest worktree capacity 20000000000 +quest worktree create example --reserve 3000000000 --ref HEAD +quest worktree status +quest worktree release example --scratch node_modules dist +quest worktree restore example +``` + +Creation prints JSON with the ID and the detached, locked checkout path under +`.worktrees/`. Release deletes only its `--scratch` paths, and refuses one that +Git does not ignore, that holds tracked files or another Git worktree, or that +is reached through a symbolic link; links inside scratch are removed, never +followed. Missing paths +are skipped, so release can be rerun. The checkout, its Git state, and +untracked files stay in place, and `status` charges what remains. Restore +reserves the checkout's disk again; then reinstall or rebuild its scratch, for +example with `just install`. + +Quest never removes a checkout. Once one is finished, remove it with +`git worktree unlock` and `git worktree remove`, which refuses modified or +untracked files but deletes ignored ones; `release` then drops its record. A +failed creation stays charged until `release` drops it. Records live in +`$QUEST_WORKTREE_HOME`, or `$XDG_STATE_HOME/quest/worktrees` (default +`~/.local/state/quest/worktrees`). No live lock is stolen: after a crash, +confirm the registry lock's owner has exited before removing it. This initial +interface refuses unknown or changed checkout ownership. + ## Work with your agents Open this repository in Claude Code or Codex. The binary carries the skills; diff --git a/quest/m0/README.md b/quest/m0/README.md index fe6546c4..7d9f254b 100644 --- a/quest/m0/README.md +++ b/quest/m0/README.md @@ -37,7 +37,7 @@ ownership-manifest design: - [Accurate landed quests](/quest/m0/landed-quests.md) - a quest-branch PR deletes or rewrites its quest, and `quest check --base` refuses one that does neither - [Expensive command gate](/quest/m0/local-gate.md) - shared CPU admission across worktrees and repositories -- [Recoverable worktrees](/quest/m0/worktree.md) - disk admission, checkpoints, release, and restore +- [Worktree process ownership](/quest/m0/worktree.md) - supervise checkout commands and release checkouts automatically - [Agent lifecycle integration](/quest/m0/worktree-agents.md) - reclaim completed and suspended checkouts through their harness - [Agent workflow helpers](/quest/m0/helpers/README.md) - CLI commands for CI waits, PR status, claims, quest moves, and serialized merges diff --git a/quest/m0/kix/runner.md b/quest/m0/kix/runner.md index f829587b..0d3c6462 100644 --- a/quest/m0/kix/runner.md +++ b/quest/m0/kix/runner.md @@ -88,4 +88,4 @@ replacing the hosted Agent SDK runtime: ## Required -- [Recoverable worktrees](/quest/m0/worktree.md) - reuse checkout supervision, checkpoints, and disk admission +- [Worktree process ownership](/quest/m0/worktree.md) - reuse checkout supervision, scratch release, and disk admission diff --git a/quest/m0/local-gate.md b/quest/m0/local-gate.md index ea43672f..1977ccd7 100644 --- a/quest/m0/local-gate.md +++ b/quest/m0/local-gate.md @@ -29,5 +29,5 @@ remain useful without multiplying full-machine builds and test stacks. ## Related -- [Recoverable worktrees](/quest/m0/worktree.md) - disk admission and checkout lifetime +- [Worktree process ownership](/quest/m0/worktree.md) - owns checkout command descendants, with its own process-ownership backend - [Own-PC runner](/quest/m0/kix/runner.md) - reuse local command supervision diff --git a/quest/m0/worktree-agents.md b/quest/m0/worktree-agents.md index 57ac09c7..804ff316 100644 --- a/quest/m0/worktree-agents.md +++ b/quest/m0/worktree-agents.md @@ -2,9 +2,10 @@ ## Goal -Quest workflows use recoverable worktree ownership through supported Codex and -Claude lifecycle integrations. A completed or suspended worker releases its -checkout without waiting for the parent chat to end or the PR to merge. +Quest workflows use `quest worktree` checkout ownership through supported +Codex and Claude lifecycle integrations. A completed or suspended worker +releases its checkout without waiting for the parent chat to end or the PR to +merge. ## Plan @@ -38,7 +39,7 @@ checkout without waiting for the parent chat to end or the PR to merge. ## Required -- [Recoverable worktrees](/quest/m0/worktree.md) - checkpoint, release, and restore +- [Worktree process ownership](/quest/m0/worktree.md) - supervised commands and automatic scratch release - [Expensive command gate](/quest/m0/local-gate.md) - resource control before removing agent-count throttles ## Related diff --git a/quest/m0/worktree.md b/quest/m0/worktree.md index a8384906..554ca376 100644 --- a/quest/m0/worktree.md +++ b/quest/m0/worktree.md @@ -1,44 +1,52 @@ -# [L] Checkpoint and release worktrees with quest worktree +# [M] Own worktree processes and release checkouts automatically ## Goal -`quest worktree` owns a recoverable checkout lifecycle: admit disk use, create -or reuse a checkout, run work, checkpoint, release, and restore on resume. -Waiting for a decision or PR review does not require retaining a large checkout. -Agent count is unrestricted by this command; expensive work uses `quest gate`. +`quest worktree` supervises the commands that run in a checkout it created, +then releases that checkout automatically once they stop by pruning its +declared scratch in place. A crash or cancellation is reconciled without losing +work, stealing a live lock, or deleting scratch a running process still uses. ## Plan -- Decided 2026-10-15: disk capacity is the admission constraint. Account for - concurrent allocations and expected build growth, with configurable limits - informed by measured footprints. A free-space check alone cannot reserve - space against simultaneous creators or subsequent builds. -- Checkpoint meaningful edits before a blocking handoff and at completion, - then release the idle checkout. An asynchronous question with independent - work still running is not a release boundary. Checkpoints are recoverable - WIP, not claims of passing tests, and do not run the validation suite. -- Preserve source, unique commits, and resume metadata independently of the - checkout before removing it. Preserve local-only submodule history outside - its disposable gitdir. Keep valuable untracked or ignored files explicitly; - never publish secrets or treat every ignored file as disposable. -- Record unresolved work and which checks passed or were not run with each - checkpoint. Recovery metadata lives outside the disposable checkout. -- Delete owned disposable scratch at handoff or exit. If scratch must live in - a checkout, ignore its declared paths and teach cleanup those same paths. - Unstaged source changes are work to preserve, not scratch to delete. -- Own foreground child processes until they stop; only then release capacity - and remove their checkout. Record ownership so crashes can be reconciled. - Reuse the owning tool for managed worktrees and report failed cleanup as a - retained allocation, not success. Do not steal live locks or remove shared - or current checkouts underneath their users. -- Preserve branches/recovery refs independently of checkout retention. A - squash-merged branch may not be an ancestor of main; exact PR-head evidence - matters when deciding whether a published result is finished. -- Verify checkpoint/restore across dirty files, unique commits, submodules, - cancellation, interrupted cleanup, and simultaneous disk admission. Measure - retained disk after complete and suspended jobs; update existing docs inline. +- Landed in [kixelated/quest#127](https://github.com/kixelated/quest/pull/127): + manual disk capacity and reservations, `create`, `release` (delete declared + scratch in place), and `restore` (reserve again; the caller rebuilds the + scratch) of Git-owned checkouts. The README section "Prunable local + checkouts" documents it as experimental and tells users to stop all work + before releasing. +- Decided (user, 2026-10-21): prune in place. Release deletes only declared + scratch that Git ignores, that holds no tracked files or other Git worktree, + and that is not reached through a symlink; it never removes the checkout or + its Git state. + Checkout-independent recovery is out of scope: four review rounds of the + checkpoint-and-remove design kept finding data-loss edge cases, while the + disk cost is scratch (2026-10-15 on this repository: 1.6 MB fresh source, + 376 MB built). +- Decided 2026-10-21 (user): this quest brings its own process-ownership + backend; `quest gate` admits on CPU load and memory only and owns no + descendants. The Linux systemd scope backend (scopes, slot handshake, + `cgroup.kill` inode pinning, prerequisite probes, opt-in systemd tests) was + dropped from the gate and is restorable from commit 5684307 in + [kixelated/quest#128](https://github.com/kixelated/quest/pull/128). Process + groups and PID files cannot prove detached descendants stopped after a + supervisor crash, so hosts without a supported backend fail explicitly + instead of guessing. +- Own foreground child processes and their descendants until they stop; only + then prune scratch and release the reservation. Record ownership so a crash + can be reconciled, including a stale registry lock whose owner has exited. + Report failed cleanup as a retained allocation, not success. +- Release automatically at lifecycle boundaries: before a blocking handoff and + at completion, once meaningful edits are committed. An asynchronous question + with independent work still running is not a release boundary. Automatic + release needs the scratch declarations without a caller; recording them at + `create` is the expected route. +- Add cancellation and crash fixtures: interrupt a supervised command with live + descendants, kill the supervisor, and reconcile. Measure retained disk after + complete and suspended jobs. +- Native agent lifecycle adoption stays in the agent integration quest. ## Related -- [Expensive command gate](/quest/m0/local-gate.md) - independent CPU admission +- [Expensive command gate](/quest/m0/local-gate.md) - independent CPU and memory admission - [Agent lifecycle integration](/quest/m0/worktree-agents.md) - native agents and workflow adoption diff --git a/src/cli/main.ts b/src/cli/main.ts index 25db658e..252a59da 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -27,6 +27,7 @@ import { type Terminal, Theme } from "./theme"; import { collect, exists, load, under } from "./tree"; import { pullRequestStatus, statusLines } from "./status"; import { waitForActivity, WaitTimeout } from "./wait"; +import { Worktrees } from "./worktree"; /** Where the command writes; the process streams, or a buffer under test. */ export interface Output { @@ -153,7 +154,7 @@ export function main(argv: string[], out: Output): number | Promise { ) .argument("[path]", "Quest to explain instead of ranking every ready quest") .option("--json", "Print a deterministic JSON array") - .option("--limit ", "Maximum number of ready quests to print", positiveInteger) + .option("--limit ", "Maximum number of ready quests to print", positiveInteger("limit")) .option("--unclaimed", "Skip quests with a claim branch or open PR, listing them on stderr") .option("--remote ", "Remote whose upstream holds the claims", "origin") .action( @@ -312,6 +313,39 @@ export function main(argv: string[], out: Output): number | Promise { ); }); + const worktree = program + .command("worktree") + .description("Reserve disk for Git checkouts and release it by pruning their scratch in place"); + const state = () => new Worktrees(); + const record = (value: unknown) => print(JSON.stringify(value)); + worktree + .command("capacity ") + .description("Set this user's shared disk budget in bytes") + .action((bytes: string) => { + state().configure(positiveInteger("bytes")(bytes)); + }); + worktree + .command("status") + .description("Print the capacity, allocated bytes, and checkouts as JSON") + .action(() => record(state().status())); + worktree + .command("create ") + .description("Create an exclusively owned detached Git worktree under .worktrees/") + .requiredOption("--reserve ", "Checkout and expected build growth in bytes", positiveInteger("bytes")) + .option("--ref ", "Initial Git revision", "HEAD") + .action((name: string, options: { reserve: number; ref: string }) => + record(state().create(root(), name, options.reserve, options.ref)), + ); + worktree + .command("release ") + .description("Delete declared scratch and keep the checkout; drop the record of a removed checkout") + .option("--scratch ", "Ignored paths without tracked files, such as dependencies and build output", []) + .action((id: string, options: { scratch: string[] }) => record(state().release(id, options.scratch))); + worktree + .command("restore ") + .description("Reserve a released checkout's disk again before rebuilding its scratch") + .action((id: string) => record(state().restore(id))); + /** Changes on stdout; mentions and check findings on stderr, which fail the command. */ const report = (result: Report): number => { for (const line of result.changes) print(line); @@ -454,11 +488,13 @@ function positiveSeconds(value: string): number { return seconds; } -/** A result limit must be a positive, safe integer. */ -function positiveInteger(value: string): number { - const count = Number(value); - if (!/^\d+$/.test(value) || !Number.isSafeInteger(count) || count <= 0) { - throw new InvalidArgumentError("limit must be a positive safe integer"); - } - return count; +/** Parses a positive, safe integer option named `name`. */ +function positiveInteger(name: string): (value: string) => number { + return (value) => { + const count = Number(value); + if (!/^\d+$/.test(value) || !Number.isSafeInteger(count) || count <= 0) { + throw new InvalidArgumentError(`${name} must be a positive safe integer`); + } + return count; + }; } diff --git a/src/cli/worktree.ts b/src/cli/worktree.ts new file mode 100644 index 00000000..7b669f2d --- /dev/null +++ b/src/cli/worktree.ts @@ -0,0 +1,360 @@ +// Git checkouts under a shared disk budget. Release prunes declared scratch in place. +import { execFileSync } from "node:child_process"; +import { randomUUID } from "node:crypto"; +import { + type Stats, + existsSync, + lstatSync, + mkdirSync, + readFileSync, + readdirSync, + realpathSync, + renameSync, + rmSync, + statSync, + statfsSync, + writeFileSync, +} from "node:fs"; +import { homedir } from "node:os"; +import { dirname, isAbsolute, join, resolve, sep } from "node:path"; + +export interface WorktreeRecord { + id: string; + name: string; + repository: string; + path: string; + reserve: number; + phase: "creating" | "active" | "released"; + failure?: string; +} +interface Registry { + version: 1; + capacity: number; + records: WorktreeRecord[]; +} + +function git(root: string, ...args: string[]): string { + return execFileSync("git", ["-C", root, ...args], { + encoding: "utf8", + maxBuffer: 64 * 1024 * 1024, + stdio: ["ignore", "pipe", "pipe"], + }); +} +function save(path: string, value: unknown): void { + const temporary = `${path}.${randomUUID()}.tmp`; + writeFileSync(temporary, `${JSON.stringify(value, null, 2)}\n`, { mode: 0o600 }); + renameSync(temporary, path); +} +/** The entry itself, without following a final symlink; undefined when it is missing. */ +function entry(path: string): Stats | undefined { + try { + return lstatSync(path); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return undefined; + throw error; + } +} +/** The filesystem holding a path, or its nearest existing ancestor. */ +function device(path: string): number { + try { + return statSync(path).dev; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT" || dirname(path) === path) throw error; + return device(dirname(path)); + } +} +/** One spelling per path: Git prints `C:/...`, Node `C:\...`, and Windows may use 8.3 names. */ +function canonical(path: string): string { + return existsSync(path) ? realpathSync.native(path) : resolve(path); +} +/** Git's registered checkouts, with canonical paths and lock reasons. */ +function checkouts(repository: string): { path: string; locked?: string }[] { + return git(repository, "worktree", "list", "--porcelain", "-z") + .split("\0\0") + .flatMap((entry) => { + const lines = entry.split("\0"); + const path = lines.find((line) => line.startsWith("worktree "))?.slice(9); + const locked = lines.find((line) => line === "locked" || line.startsWith("locked "))?.slice(7); + return path ? [{ path: canonical(path), locked }] : []; + }); +} +function childPath(path: string): string { + if ( + !path || + isAbsolute(path) || + path.split(/[\\/]/).some((part) => part === ".." || part.toLowerCase() === ".git") || + resolve("/", path) === "/" + ) { + throw new Error(`expected a relative path without .. or .git: ${path}`); + } + return path.replaceAll("\\", "/").replace(/\/$/, ""); +} +/** Whether a scratch path exists; refuses one reached through a symlink, which could leave the checkout. */ +function present(root: string, path: string): boolean { + const parts = path.split("/"); + for (let i = 1; i <= parts.length; i++) { + const stat = entry(join(root, ...parts.slice(0, i))); + if (!stat) return false; + if (i < parts.length && !stat.isDirectory()) + throw new Error(`scratch path is reached through a symlink or file: ${path}`); + } + return true; +} +/** The paths Git ignores; check-ignore exits 1 when it matches none. */ +function ignored(root: string, paths: string[]): Set { + try { + const output = execFileSync("git", ["-C", root, "check-ignore", "-z", "--stdin"], { + encoding: "utf8", + input: `${paths.join("\0")}\0`, + stdio: ["pipe", "pipe", "pipe"], + }); + return new Set(output.split("\0")); + } catch (error) { + if ((error as { status?: number }).status === 1) return new Set(); + throw error; + } +} + +/** The first linked worktree of any repository, marked by a `.git` file, at or under `path`; symlinks are never followed. */ +function linkedWorktree(path: string): string | undefined { + if (!entry(path)?.isDirectory()) return undefined; + if (entry(join(path, ".git"))?.isFile()) return path; + for (const child of readdirSync(path)) { + const found = child === ".git" ? undefined : linkedWorktree(join(path, child)); + if (found) return found; + } + return undefined; +} + +/** + * Git's lock reason for the checkout actually at `path`, read through its own `.git` file and + * administrative directory, so a root replaced by another checkout reports that checkout's lock. + */ +function lockReason(path: string): string | undefined { + try { + const admin = /^gitdir: (.+)$/m.exec(readFileSync(join(path, ".git"), "utf8"))?.[1]; + return admin && readFileSync(resolve(path, admin.trim(), "locked"), "utf8").replace(/\r?\n$/, ""); + } catch (error) { + if (["ENOENT", "ENOTDIR", "EISDIR"].includes((error as NodeJS.ErrnoException).code ?? "")) return undefined; + throw error; + } +} + +/** Allocated blocks, including ignored build output; symlinks are never traversed and hard links count once. */ +export function diskUsage(path: string, seen = new Set()): number { + const stat = entry(path); + if (!stat) return 0; + if (stat.nlink > 1 && !stat.isDirectory()) { + const inode = `${stat.dev}:${stat.ino}`; + if (seen.has(inode)) return 0; + seen.add(inode); + } + // Allocated blocks, not logical length, so sparse files are not overcharged. Windows reports no blocks. + let bytes = process.platform === "win32" ? stat.size : stat.blocks * 512; + if (stat.isDirectory()) for (const child of readdirSync(path)) bytes += diskUsage(join(path, child), seen); + return bytes; +} + +/** All allocations share one registry across repositories for this user/host. */ +export class Worktrees { + readonly home: string; + constructor( + home = process.env.QUEST_WORKTREE_HOME ?? + join(process.env.XDG_STATE_HOME ?? join(homedir(), ".local", "state"), "quest", "worktrees"), + ) { + mkdirSync(home, { recursive: true, mode: 0o700 }); + this.home = realpathSync(home); + } + private locked(action: (registry: Registry) => T): T { + const lock = join(this.home, "lock"); + try { + mkdirSync(lock, { mode: 0o700 }); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "EEXIST") + throw new Error( + `worktree registry is locked: ${lock}; never remove a live lock; after a crash inspect owner.json and reconcile it explicitly`, + ); + throw error; + } + try { + save(join(lock, "owner.json"), { pid: process.pid, started: new Date().toISOString() }); + const file = join(this.home, "registry.json"); + const registry: Registry = existsSync(file) + ? JSON.parse(readFileSync(file, "utf8")) + : { version: 1, capacity: 0, records: [] }; + if (registry.version !== 1) throw new Error("unsupported worktree registry version"); + try { + return action(registry); + } finally { + save(file, registry); + } + } finally { + rmSync(lock, { recursive: true }); + } + } + private find(registry: Registry, id: string): WorktreeRecord { + const record = registry.records.find((item) => item.id === id || item.name === id); + if (!record) throw new Error(`unknown worktree: ${id}`); + return record; + } + /** + * Bytes on disk, plus the growth active reservations still expect. Released checkouts keep only + * what remains. `local` is the expected growth on the filesystem holding `parent`. + */ + private usage(registry: Registry, parent?: string): { used: number; growth: number; local: number } { + const target = parent === undefined ? undefined : device(parent); + let used = 0; + let growth = 0; + let local = 0; + for (const record of registry.records) { + const bytes = diskUsage(record.path); + used += bytes; + if (record.phase === "released") continue; + const expected = Math.max(0, record.reserve - bytes); + growth += expected; + if (target !== undefined && device(record.path) === target) local += expected; + } + return { used, growth, local }; + } + private admit(registry: Registry, reserve: number, parent: string): void { + const { used, growth, local } = this.usage(registry, parent); + if (used + growth + reserve > registry.capacity) + throw new Error( + `disk capacity exceeded: ${used + growth} allocated + ${reserve} requested > ${registry.capacity}`, + ); + const fs = statfsSync(parent); + if (Number(fs.bavail) * Number(fs.bsize) < local + reserve) + throw new Error("insufficient free space after accounting for concurrent build reservations"); + } + /** Capacity persists; reducing it below existing allocations is rejected. */ + configure(capacity: number): void { + this.locked((registry) => { + if (!Number.isSafeInteger(capacity) || capacity <= 0) + throw new Error("capacity must be a positive safe integer in bytes"); + const { used, growth } = this.usage(registry); + if (used + growth > capacity) throw new Error("capacity is below existing allocations"); + registry.capacity = capacity; + }); + } + status(): { capacity: number; allocated: number; records: WorktreeRecord[] } { + return this.locked((registry) => { + const { used, growth } = this.usage(registry); + return { capacity: registry.capacity, allocated: used + growth, records: registry.records }; + }); + } + create(root: string, name: string, reserve: number, ref = "HEAD"): WorktreeRecord { + return this.locked((registry) => { + if (!/^[a-zA-Z0-9][a-zA-Z0-9._-]*$/.test(name)) + throw new Error("name must contain only letters, digits, dots, underscores, or hyphens"); + if (!Number.isSafeInteger(reserve) || reserve <= 0) + throw new Error("reserve must be a positive safe integer in bytes, including expected build growth"); + if (registry.records.some((item) => item.name === name)) + throw new Error(`worktree name already exists; use restore or another name: ${name}`); + const repository = canonical(git(root, "rev-parse", "--show-toplevel").trim()); + const path = join(repository, ".worktrees", name); + mkdirSync(dirname(path), { recursive: true }); + if (existsSync(path)) throw new Error(`refusing to adopt an existing checkout: ${path}`); + this.admit(registry, reserve, dirname(path)); + const record: WorktreeRecord = { id: randomUUID(), name, repository, path, reserve, phase: "creating" }; + registry.records.push(record); + save(join(this.home, "registry.json"), registry); // Reservation survives a crash in git. + try { + git( + repository, + "worktree", + "add", + "--detach", + "--lock", + "--reason", + `quest worktree ${record.id}`, + path, + ref, + ); + record.phase = "active"; + return record; + } catch (error) { + record.failure = String(error); + throw error; + } + }); + } + /** Refuses unless this record exclusively owns the checkout; false once Git and the disk both lack it. */ + private owned(record: WorktreeRecord): boolean { + const path = canonical(record.path); + const registered = checkouts(record.repository).find((entry) => entry.path === path); + if (!registered && !existsSync(path)) return false; + // Canonical paths follow a replaced root to its target, so ask the checkout on disk for its lock. + const lock = existsSync(path) ? lockReason(record.path) : registered?.locked; + if (lock !== `quest worktree ${record.id}`) + throw new Error( + "checkout ownership/lock changed; use the owning tool, never adopt or remove a shared checkout", + ); + return true; + } + /** + * Deletes only the declared scratch, which Git must ignore and which holds no tracked files. + * The checkout, its Git state, and untracked source stay in place. A checkout that Git and the + * disk both lack, such as one removed with `git worktree remove`, drops its record instead. + */ + release(id: string, scratch: string[]): WorktreeRecord { + return this.locked((registry) => { + const record = this.find(registry, id); + if (!this.owned(record)) { + registry.records.splice(registry.records.indexOf(record), 1); + return record; + } + if (record.phase === "creating") + throw new Error( + `creation was interrupted; inspect and remove the partial checkout with git worktree remove, then release again: ${record.path}`, + ); + const root = canonical(record.path); + const paths = scratch.map(childPath).filter((path) => present(root, path)); // Reruns skip pruned paths. + const admin = entry(join(root, ".git")); + for (const path of paths) { + // Case folding and other platform aliases can still name the checkout's .git entry. + const stat = entry(join(root, path)); + if (admin && stat?.dev === admin.dev && stat.ino === admin.ino) + throw new Error(`scratch path is the checkout's .git entry: ${path}`); + } + if (paths.length) { + // icase, because a case-insensitive filesystem resolves `CACHE` to a tracked `cache`. + const pathspecs = paths.map((path) => `:(literal,icase)${path}`); + const tracked = git(root, "ls-files", "-z", "--", ...pathspecs) + .split("\0") + .filter(Boolean); + if (tracked.length) + throw new Error(`scratch includes tracked files: ${tracked.slice(0, 20).join(", ")}`); + const matched = ignored(root, paths); + const unignored = paths.filter((path) => !matched.has(path)); + if (unignored.length) throw new Error(`scratch must be ignored by Git: ${unignored.join(", ")}`); + // Another checkout, such as one created inside this one, is work rather than scratch. + const registered = checkouts(record.repository) + .map((checkout) => checkout.path) + .filter((path) => path !== root); + for (const path of paths) { + const target = canonical(join(root, path)); + const nested = + registered.find((checkout) => checkout === target || checkout.startsWith(target + sep)) ?? + linkedWorktree(join(root, path)); + if (nested) throw new Error(`scratch contains a Git worktree: ${path} holds ${nested}`); + } + // Removal unlinks symlinks rather than following them; force skips paths an overlapping parent removed. + for (const path of paths) rmSync(join(root, path), { recursive: true, force: true }); + } + record.phase = "released"; + return record; + }); + } + /** Reserves the released checkout's disk again; the caller then reinstalls or rebuilds its scratch. */ + restore(id: string): WorktreeRecord { + return this.locked((registry) => { + const record = this.find(registry, id); + if (!this.owned(record)) throw new Error(`checkout is missing; release drops its record: ${record.path}`); + if (record.phase === "active") return record; + if (record.phase !== "released") throw new Error(`cannot restore a ${record.phase} checkout`); + this.admit(registry, Math.max(0, record.reserve - diskUsage(record.path)), dirname(record.path)); + record.phase = "active"; + return record; + }); + } +} diff --git a/test/worktree.test.ts b/test/worktree.test.ts new file mode 100644 index 00000000..77c7727e --- /dev/null +++ b/test/worktree.test.ts @@ -0,0 +1,243 @@ +import { execFileSync, spawn } from "node:child_process"; +import { + appendFileSync, + existsSync, + linkSync, + lstatSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + symlinkSync, + truncateSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { Worktrees, diskUsage } from "../src/cli/worktree"; +import { run } from "./fixture"; + +const roots: string[] = []; +function git(root: string, ...args: string[]): string { + return execFileSync("git", ["-C", root, ...args], { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }).trim(); +} +function repository(path: string): string { + mkdirSync(path, { recursive: true }); + git(path, "init", "-b", "main"); + git(path, "config", "user.name", "Fixture"); + git(path, "config", "user.email", "fixture@localhost"); + writeFileSync(join(path, "file"), "original\n"); + writeFileSync(join(path, ".gitignore"), "cache/\n.env\n.worktrees/\n"); + git(path, "add", "."); + git(path, "commit", "-m", "initial"); + return path; +} +function fixture() { + const root = mkdtempSync(join(tmpdir(), "quest-worktree-test-")); + roots.push(root); + const repo = repository(join(root, "repo")); + const home = join(root, "state"); + const worktrees = new Worktrees(home); + worktrees.configure(100_000_000); + return { root, repo, home, worktrees }; +} +afterEach(() => { + vi.unstubAllEnvs(); + for (const root of roots.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +// Cases run up to 20 real Git processes. Process creation on the Windows ARM runner +// costs ~80-250 ms while other test files start, so the 5 s unit-test default is too short. +describe("worktrees", { timeout: 20_000 }, () => { + it("prunes declared scratch in place and keeps source, Git state, and untracked files", () => { + const { repo, worktrees } = fixture(); + const record = worktrees.create(repo, "work", 10_000_000); + writeFileSync(join(record.path, "file"), "committed\n"); + git(record.path, "commit", "-am", "local only"); + const head = git(record.path, "rev-parse", "HEAD"); + writeFileSync(join(record.path, "file"), "staged\n"); + git(record.path, "add", "file"); + writeFileSync(join(record.path, "file"), "unstaged\n"); + writeFileSync(join(record.path, "new-source"), "untracked source"); + writeFileSync(join(record.path, ".env"), "ignored and undeclared"); + mkdirSync(join(record.path, "cache", "nested"), { recursive: true }); + writeFileSync(join(record.path, "cache", "nested", "artifact"), "disposable"); + const status = git(record.path, "status", "--porcelain=v1"); + expect(worktrees.release(record.id, ["cache/", "cache/nested", "missing"]).phase).toBe("released"); + expect(existsSync(join(record.path, "cache"))).toBe(false); + expect(git(record.path, "status", "--porcelain=v1")).toBe(status); + expect(git(record.path, "rev-parse", "HEAD")).toBe(head); + expect(git(record.path, "show", ":file")).toBe("staged"); + expect(readFileSync(join(record.path, "file"), "utf8")).toBe("unstaged\n"); + expect(readFileSync(join(record.path, "new-source"), "utf8")).toBe("untracked source"); + expect(readFileSync(join(record.path, ".env"), "utf8")).toBe("ignored and undeclared"); + expect(worktrees.restore(record.id).phase).toBe("active"); + }); + it("refuses scratch that is tracked, not ignored, or outside the checkout", () => { + const { repo, worktrees } = fixture(); + const record = worktrees.create(repo, "work", 10_000_000); + writeFileSync(join(record.path, "new"), "source"); + mkdirSync(join(record.path, "cache")); + writeFileSync(join(record.path, "cache", "forced"), "tracked despite the ignore rule"); + git(record.path, "add", "-f", "cache/forced"); + expect(() => worktrees.release(record.id, ["file"])).toThrow("tracked files: file"); + expect(() => worktrees.release(record.id, ["cache"])).toThrow("tracked files: cache/forced"); + // Another spelling, which a case-insensitive filesystem resolves to the tracked directory. + mkdirSync(join(record.path, "CACHE"), { recursive: true }); + appendFileSync(join(repo, ".git", "info", "exclude"), "/CACHE/\n"); + expect(() => worktrees.release(record.id, ["CACHE"])).toThrow("tracked files: cache/forced"); + expect(() => worktrees.release(record.id, ["new"])).toThrow("ignored by Git: new"); + expect(() => worktrees.release(record.id, ["../outside"])).toThrow("relative path"); + expect(() => worktrees.release(record.id, [".git"])).toThrow("relative path"); + expect(() => worktrees.release(record.id, [".GIT"])).toThrow("relative path"); + // Any other alias of the .git entry, here a hard link that an ignore rule matches. + linkSync(join(record.path, ".git"), join(record.path, ".env")); + expect(() => worktrees.release(record.id, [".env"])).toThrow(".git entry: .env"); + expect(readFileSync(join(record.path, "new"), "utf8")).toBe("source"); + expect(readFileSync(join(record.path, "cache", "forced"), "utf8")).toBe("tracked despite the ignore rule"); + expect(worktrees.status().records[0].phase).toBe("active"); + }); + it.skipIf(process.platform === "win32")("never follows symlinks out of the checkout", () => { + const { root, repo, worktrees } = fixture(); + const outside = join(root, "outside"); + mkdirSync(join(outside, "cache"), { recursive: true }); + writeFileSync(join(outside, "cache", "keep"), "outside"); + const record = worktrees.create(repo, "work", 10_000_000); + symlinkSync(outside, join(record.path, "linked")); + expect(() => worktrees.release(record.id, ["linked/cache"])).toThrow("reached through a symlink"); + mkdirSync(join(record.path, "cache")); + symlinkSync(outside, join(record.path, "cache", "link")); + symlinkSync(outside, join(record.path, ".env")); + worktrees.release(record.id, ["cache", ".env"]); + expect(existsSync(join(record.path, "cache"))).toBe(false); + expect(existsSync(join(record.path, ".env"))).toBe(false); + expect(readFileSync(join(outside, "cache", "keep"), "utf8")).toBe("outside"); + }); + it.skipIf(process.platform === "win32")("charges allocated blocks of dangling symlinks and sparse files", () => { + const { root } = fixture(); + const dangling = join(root, "dangling"); + symlinkSync("missing", dangling); + expect(diskUsage(dangling)).toBe(lstatSync(dangling).blocks * 512); + const sparse = join(root, "sparse"); + writeFileSync(sparse, ""); + truncateSync(sparse, 1 << 30); + expect(diskUsage(sparse)).toBeLessThan(1 << 20); + const linked = join(root, "linked"); + mkdirSync(linked); + writeFileSync(join(linked, "a"), Buffer.alloc(1 << 16, 1)); + linkSync(join(linked, "a"), join(linked, "b")); + expect(diskUsage(linked)).toBe((lstatSync(linked).blocks + lstatSync(join(linked, "a")).blocks) * 512); + }); + it("charges active reservations and the footprint released checkouts keep", () => { + const { repo, worktrees } = fixture(); + const reserve = 1_000_000; + worktrees.configure(reserve); + const record = worktrees.create(repo, "work", reserve); + expect(worktrees.status().allocated).toBe(reserve); + worktrees.release(record.id, []); + const retained = worktrees.status().allocated; + expect(retained).toBeGreaterThan(0); + expect(retained).toBeLessThan(reserve); + worktrees.create(repo, "other", reserve - retained); + expect(() => worktrees.restore(record.id)).toThrow("disk capacity exceeded"); + expect(worktrees.status().records[0].phase).toBe("released"); + }); + it("drops the record once its checkout is removed", () => { + const { repo, worktrees } = fixture(); + const record = worktrees.create(repo, "work", 10_000_000); + expect(() => worktrees.release(record.id, ["missing"])).not.toThrow(); + git(repo, "worktree", "unlock", record.path); + git(repo, "worktree", "remove", record.path); + worktrees.release(record.id, []); + expect(worktrees.status().records).toEqual([]); + expect(() => worktrees.restore(record.id)).toThrow("unknown worktree"); + expect(worktrees.create(repo, "work", 10_000_000).phase).toBe("active"); + }); + it("serializes reservations across independent repositories and preserves failed allocations", () => { + const { root, repo, home, worktrees } = fixture(); + const second = repository(join(root, "second")); + worktrees.configure(20_000_000); + worktrees.create(repo, "first", 12_000_000); + expect(() => new Worktrees(home).create(second, "second", 12_000_000)).toThrow("disk capacity exceeded"); + expect(() => worktrees.configure(1_000_000)).toThrow("below existing"); + mkdirSync(join(home, "lock")); + expect(() => worktrees.status()).toThrow("never remove a live lock"); + rmSync(join(home, "lock"), { recursive: true }); + expect(() => worktrees.create(second, "failed", 1_000_000, "nonexistent-ref")).toThrow(); + expect(worktrees.status().records.find((record) => record.name === "failed")?.phase).toBe("creating"); + worktrees.release("failed", []); + expect(worktrees.status().records.map((record) => record.name)).toEqual(["first"]); + }); + it("admits simultaneous creators atomically without oversubscribing disk", async () => { + const { root, repo, home, worktrees } = fixture(); + const second = repository(join(root, "second")); + worktrees.configure(15_000_000); + const module = new URL("../src/cli/worktree.ts", import.meta.url).href; + const create = (repository: string, name: string) => + new Promise((resolve, reject) => { + const source = `import { Worktrees } from ${JSON.stringify(module)}; new Worktrees(${JSON.stringify(home)}).create(${JSON.stringify(repository)}, ${JSON.stringify(name)}, 10_000_000);`; + const child = spawn( + process.execPath, + ["--experimental-strip-types", "--input-type=module", "-e", source], + { stdio: "ignore" }, + ); + child.on("error", reject); + child.on("exit", resolve); + }); + const results = await Promise.all([create(repo, "one"), create(second, "two")]); + expect(results.filter((result) => result === 0)).toHaveLength(1); + expect(worktrees.status().records).toHaveLength(1); + expect(worktrees.status().allocated).toBeLessThanOrEqual(15_000_000); + }); + it("refuses foreign checkouts and changed ownership", () => { + const { repo, worktrees } = fixture(); + mkdirSync(join(repo, ".worktrees", "foreign"), { recursive: true }); + expect(() => worktrees.create(repo, "foreign", 10_000_000)).toThrow("refusing to adopt"); + const record = worktrees.create(repo, "work", 10_000_000); + git(repo, "worktree", "unlock", record.path); + git(repo, "worktree", "lock", "--reason", "another owner", record.path); + expect(() => worktrees.release(record.id, [])).toThrow("ownership/lock changed"); + expect(() => worktrees.restore(record.id)).toThrow("ownership/lock changed"); + }); + it("refuses scratch that contains another Git worktree", () => { + const { root, repo, worktrees } = fixture(); + const outer = worktrees.create(repo, "outer", 10_000_000); + // Created from inside the outer checkout, so it lands in the outer checkout's ignored .worktrees/. + const inner = worktrees.create(outer.path, "inner", 10_000_000); + expect(inner.path).toBe(join(outer.path, ".worktrees", "inner")); + expect(() => worktrees.release(outer.id, [".worktrees"])).toThrow("scratch contains a Git worktree"); + expect(existsSync(join(inner.path, "file"))).toBe(true); + // A linked worktree of an unrelated repository, such as an agent's checkout under an ignored directory. + const foreign = repository(join(root, "foreign")); + git(foreign, "worktree", "add", "--detach", join(outer.path, "cache", "agent")); + expect(() => worktrees.release(outer.id, ["cache"])).toThrow("scratch contains a Git worktree: cache"); + expect(existsSync(join(outer.path, "cache", "agent", "file"))).toBe(true); + expect(worktrees.status().records.map((record) => record.phase)).toEqual(["active", "active"]); + }); + it.skipIf(process.platform === "win32")("refuses a root replaced by a link to another checkout", () => { + const { repo, worktrees } = fixture(); + const replaced = worktrees.create(repo, "first", 10_000_000); + const other = worktrees.create(repo, "second", 10_000_000); + mkdirSync(join(other.path, "cache")); + rmSync(replaced.path, { recursive: true }); + symlinkSync(other.path, replaced.path); + expect(() => worktrees.release(replaced.id, ["cache"])).toThrow("ownership/lock changed"); + expect(existsSync(join(other.path, "cache"))).toBe(true); + }); + it("wires capacity, reservation, and scratch through the command line", () => { + const { repo, home } = fixture(); + vi.stubEnv("QUEST_WORKTREE_HOME", home); + expect(run("worktree", "create", "cli", "--root", repo).code).toBe(2); + expect(run("worktree", "capacity", "0").code).toBe(2); + const created = run("--root", repo, "worktree", "create", "cli", "--reserve", "10000000"); + expect(created.code).toBe(0); + const { path } = JSON.parse(created.stdout); + mkdirSync(join(path, "cache")); + expect(run("worktree", "release", "cli", "--scratch", "cache").code).toBe(0); + expect(existsSync(join(path, "cache"))).toBe(false); + const status = run("worktree", "status"); + expect(status.code).toBe(0); + expect(JSON.parse(status.stdout).records[0].phase).toBe("released"); + }); +});