diff --git a/README.md b/README.md index bb665535..c7b5939b 100644 --- a/README.md +++ b/README.md @@ -1592,17 +1592,14 @@ prompt or reply. Since another repository may use the same branch name, the session must also be this repository's, as the repositories token-max resolved for it on its own host name, or, once its checkout is deleted, those resolved there for another session in that exact directory, as a Codex log records -(never a parent's, which a nested clone may not share). A sub-agent or -spawned session that names no PR's branch, as one in a detached checkout, -takes the PR of the session that spawned it, unless its own repository is -known to be another; otherwise it stays -unattributed. A session is classifying when the PR's `## Review` record names -it as the [risk classifier](#risk-tiers), reviewing when the record names it as -a reviewer or Codex marks it a review, its parent's role when both are on the -same PR, so a review's sub-agents count as reviewing, implementing when its -first request since `--since` came before the PR opened, and iterating -otherwise, so a session -that opened a PR before `--since` and kept working on it reads as iterating. Each PR shows its `## Models` size, its risk tier and +(never a parent's, which a nested clone may not share); otherwise it stays +unattributed. A session on no such branch, such as a sub-agent in a detached +checkout, belongs to its parent's PRs unless its own repository is known to be +another, and a coordinator whose sub-agents worked on several PRs belongs to all +of them, its cost counting equally toward each, so a PR's sessions add up to its +whole cost. Each session gets the record +[`quest records sender`](#session-records) would send, and its role comes from +it, so a review's sub-agents count as reviewing. Each PR shows its `## Models` size, its risk tier and whether a reviewer raised it, and its implementers, and as `unread:` each Implementer line it cannot read; each reviewer the `quest review` record names, with its latest round and score and its tally of findings across every round, marked `shadow` for the shadow, where a merged PR whose last round left only low @@ -1662,7 +1659,8 @@ Sessions on no PR here, from any repository or agent token-max reads, total on an `Unattributed` line. `--pr` and `--quest` narrow the report to one PR or one quest's branch, and `--json` prints it as one object. Without `token-max` on `PATH`, it says how to install -it and reports the PRs' records alone. +it and reports the PRs' records alone. Each PR's `waits:` line sums its +sessions' waits by kind, a shared session's in its share. `--ssh HOST`, repeatable, also reads that SSH destination's sessions through the token-max installed there, matching each through the repositories resolved @@ -1675,6 +1673,101 @@ unfinished, is listed under `Lower bounds` with its unreadable lines, and every cost it reaches reads `or more`. The land fixer's container keeps no session log, so its fixes show without tokens. +### Session records + +When an agent session's turn ends in a checkout that uses Quest, `quest` +sends Kixel Code one record of it, rebuilt each time, so a resend replaces +the last and the store follows the session as it grows. The store keeps the +latest upload, since one sender sends a session's uploads in order. The type is +`SessionRecord`, exported from `quest/core` with `parseRecord` and +`parseUpload`, which refuse a malformed record whole: + +```json +{"harness":"claude","session":"7ef44bda-fb9b-436f-9ed8-9013d0f38d46","parent":null,"repository":"acme/app","branch":"quest/m0/login","quest":"quest/m0/login.md","pulls":[42],"role":"implementing","models":[{"model":"claude-opus-5-5","effort":"high","input":230,"cacheWrite":211500,"cacheRead":15800000,"output":88800,"cost":6.64,"complete":true}],"cost":6.64,"complete":true,"start":"2026-11-14T00:06:12.270Z","end":"2026-11-14T00:31:40.120Z","waits":[{"kind":"gate-queue","start":"2026-11-14T00:22:07.386Z","end":"2026-11-14T00:22:07.402Z","until":null}]} +``` + +- `pulls` are the PRs it worked on, attributed as `quest usage` attributes + them; a session with none yet, such as planning before its PR opens, keeps + its `branch`, and the next record names the PR. +- `role` is classifying when a PR's `## Review` record names it as the + [risk classifier](#risk-tiers), reviewing when it names it as a reviewer or + Codex marks it a review, else the one the first quest skill it ran gives: planning + (`/quest-plan`, `/quest-audit`, `/quest-recurring`, `/quest-import`, + `/quest-grill`, `/quest-delete`, `/quest-export`), implementing + (`/quest-start`), iterating (`/quest-iterate`), reviewing (`/quest-review`), + merging (`/quest-merge`, `/quest-complete`), or coordinating (`/quest-run`, + `/quest-spawn`). Without one, a sub-agent takes its parent's role unless that + is coordinating; a session on its sub-agents' PRs only is coordinating; one + on a `quest-plan/`, `quest-audit/`, `quest-recurring/`, or `quest-import/` + branch is planning; and the rest are implementing when their first request + came before the PR opened, else iterating. +- `waits` holds each wait a tool timed: `gate-queue` and `gate-run` for an + outermost `quest gate`, `pull` for `quest wait` with the first activity it + printed as `until` (null on timeout), `merge-queue` for a `quest wait` that + ended on `merged` or `removed_from_merge_queue`, and `person` from a blocking + question, such as `AskUserQuestion` or `request_user_input`, to its answer. + `quest gate` and `quest wait` log theirs under + `$XDG_STATE_HOME/quest/waits` (`~/.local/state`), one file per day kept 90 + days, whenever `CLAUDE_CODE_SESSION_ID` or `CODEX_THREAD_ID` names the + session; a Claude sub-agent's commands carry its parent's ID, so a wait goes + to the one sub-agent of it running then in the wait's directory. Waits may + overlap, as parallel gates do. `null` means they could not be read, as for a + session on another host in `quest usage --ssh`, or one whose transcript has + lines token-max could not read either. Like token costs, waits count only + those that ended since the window's start. + +The record holds no prompt, reply, or transcript text: skills and questions +are read from the transcript on this host, and only their names and times +leave it. + +A checkout names its endpoint in a committed `.quest.json`: + +```json +{ "records": "https://kixel.example/records" } +``` + +and each user pairs a token with it in `$XDG_CONFIG_HOME/quest/records.json` +(`~/.config`), so a checkout naming another endpoint, as a hostile branch +might, is never sent the token or a record: + +```json +{ "https://kixel.example/records": "token" } +``` + +The endpoint must be HTTPS, or HTTP to `localhost`. A checkout naming none, or +an endpoint the user holds no token for, sends nothing, and the hook says so +once as a system message. + +The `Stop` hook [`quest setup`](#configure-your-agent) installs for Claude Code +and Codex only queues the session under `$XDG_STATE_HOME/quest/records`. One +sender per user, `quest records sender`, sends the queue each minute at low +priority, so a busy host's stops never spend its GitHub quota. Its service +manager runs exactly one, so passes never overlap and a session's uploads reach +the store in order: + +```console +$ quest records install +installed and started /home/me/.config/systemd/user/quest-records.service +``` + +`quest records install` writes a systemd user service on Linux or a launchd +agent on macOS running `quest records sender` with the current `PATH`, and +(re)starts it; rerun it after upgrading `quest`. `quest records uninstall` +removes it. While no sender runs, stops still queue, and the next stop says so +once per outage. + +Each pass runs token-max on this host from the earliest queued session's first +instant, reads the PRs updated since then, and posts `{"records": [...]}` with +`Authorization: Bearer `, one upload per endpoint and repository, each +queued session with its sub-agents, which stop without the hook naming them. +Each upload is checked with `parseUpload` before it is posted. A 2xx drops the +sessions from the queue, unless the hook queued one again meanwhile. A 401 or +403 drops them and sends that endpoint nothing more until the token changes. +Any other failure keeps them queued, and the next stop says so once. A failing +endpoint and repository back off, waiting a minute and doubling up to six +hours, and sit out the passes meanwhile, so they never widen the others' +window. token-max missing is retried each pass. + ### Typed messages `quest messages` prints the messages a person typed to agents on this host @@ -2048,7 +2141,9 @@ background commands when it ends, and a session that ends its turn on one leaves its result unread. A command is the agent's when its transcript holds the ID Claude Code gave it on backgrounding it, so a sibling's gate does not hold an agent. Claude Code lists a command only while it runs, so the -refusal ends with the gate. Codex runs no such hook. +refusal ends with the gate. Codex runs no such hook. Once the turn can end, +Claude Code's and Codex's `Stop` hooks queue the session's +[record](#session-records). `quest` must be on the `PATH` the agent starts with, as for the worktree `--body`, a `body=` field, `git commit -m`, `--body-file -` with a here-document, or a body file. A file diff --git a/quest/m0/benchmark/breakage.md b/quest/m0/benchmark/breakage.md index c37b4fb0..e7602646 100644 --- a/quest/m0/benchmark/breakage.md +++ b/quest/m0/benchmark/breakage.md @@ -45,11 +45,6 @@ Agent's choices, for the implementer to revisit: - Keep names of customers and private repositories out of anything this public repository's records or PRs show. -## Required - -- [Send each session's record to Kixel Code](/quest/m0/benchmark/records.md) - - the record type and the sending path breakage records reuse - ## Related - [Mistake audit](/quest/m0/mistake-audit.md) - classifies each escape as diff --git a/quest/m0/benchmark/records-endpoint.md b/quest/m0/benchmark/records-endpoint.md new file mode 100644 index 00000000..0f184f2b --- /dev/null +++ b/quest/m0/benchmark/records-endpoint.md @@ -0,0 +1,31 @@ +# [XS] Name Kixel Code's record endpoint in each repository + +## Goal + +Every repository whose sessions Kixel Code should count commits a +`.quest.json` naming the record endpoint Kixel Code serves, so `quest`'s +stop hook sends each session's record there, and the maintainer's hosts +hold a token paired with it in `~/.config/quest/records.json`. One real +session on a moq-dev/quest PR then shows in Kixel Code's store, its +record equal to what `quest usage --json` prints for it. + +## Plan + +`quest` sends the records since +[moq-dev/quest#718](https://github.com/moq-dev/quest/pull/718): it posts +`{"records": [...]}`, each a `SessionRecord` from `quest/core`, with the +user's token as a bearer token, and takes a 401 or 403 as no access. + +- Commit `.quest.json` here first, then in each repository the benchmark + covers, with the endpoint's final URL. +- Put the token on each host, and start its sender with `quest records + install`, through kixel-ai/dotfiles, which owns the + maintainer's host configuration, not by hand. + +## Waiting + +- kixel-ai/kixel.quest: its record store serves an endpoint that stores + `quest`'s upload with `parseUpload`, keeps each session's latest upload, + refuses a token it did not issue + with 401 or 403, and it says how a person gets a token; check that its + `quest/m0/records.md` is gone from its tree diff --git a/quest/m0/benchmark/records.md b/quest/m0/benchmark/records.md deleted file mode 100644 index 7b9eb042..00000000 --- a/quest/m0/benchmark/records.md +++ /dev/null @@ -1,53 +0,0 @@ -# [L] Send each session's record to Kixel Code - -## Goal - -When an agent session on a quest or PR ends, `quest` sends one record of -it to Kixel Code: the repository, PR, and quest; its role (planning, -implementing, iterating, reviewing, merging); harness, model, and effort; -tokens and their price; wall time; and each wait a tool already times -exactly, by kind: queued in `quest gate`, running a gated command, waiting -on CI or a reviewer in `quest wait`, waiting in the merge queue, and -waiting on a person's answer. A PR's records add up to its whole cost, -including sub-agent and coordinator sessions that `quest usage` could not -trace to a PR on 2026-11-11, when it traced 28% of host spend. - -The record is a type `quest` exports, so Kixel Code, which uses `quest` as -a library, stores exactly what is sent. `quest usage` prints the same -records for a contributor's own sessions. A contributor whose checkout -names no Kixel Code endpoint, or who has no access to it, sends nothing, -and `quest` says so once. - -Non-goals: logging every command; sending prompts, replies, or any -transcript text; CI runner compute, which the forge reports and Kixel Code -reads itself. - -## Plan - -Decided 2026-11-13 (user, /quest-plan): Kixel Code stores the records, as -[the epic](/quest/m0/benchmark/README.md) records, and measuring covers -"the time spent blocked on GitHub or CI or git or a human". - -Agent's choices, for the implementer to revisit: - -- Build the record from what `quest usage` already reads per session, so - one code path prices a session for both. Its attribution through - token-max is - [moq-dev/quest#704](https://github.com/moq-dev/quest/pull/704). -- Send from the hook that already runs when a session stops, and resend - what failed on the next run, so a host that is offline loses nothing. A - record carries the session's id, so a resend replaces and never doubles. -- A wait on a person is the time between an agent's question and the - typed answer, both timestamped in the transcript; `quest messages` - already pairs them. -- Sessions with no PR yet, such as planning, are sent against their branch - and joined to the PR once it opens. -- Leave time no tool times, such as fighting git or redoing work, to - [the time split](/quest/m0/benchmark/time-split.md). - -## Related - -- [Development as a benchmark](/quest/m0/benchmark/README.md) - the - decisions this builds on -- [Price sessions through token-max](https://github.com/moq-dev/quest/pull/704) - - the per-session pricing these records reuse diff --git a/quest/m0/benchmark/time-split.md b/quest/m0/benchmark/time-split.md index f7f521ed..87d251c8 100644 --- a/quest/m0/benchmark/time-split.md +++ b/quest/m0/benchmark/time-split.md @@ -42,11 +42,6 @@ Agent's choices, for the implementer to revisit: the categories and the one-line note leave the host; a note names no customer and quotes no message. -## Required - -- [Send each session's record to Kixel Code](/quest/m0/benchmark/records.md) - - the record the split is added to, and the exact waits it excludes - ## Related - [Mistake audit](/quest/m0/mistake-audit.md) - groups what went wrong diff --git a/src/cli/claims.ts b/src/cli/claims.ts index 92bce918..412a5f02 100644 --- a/src/cli/claims.ts +++ b/src/cli/claims.ts @@ -72,9 +72,7 @@ export async function upstream(root: string, remote: Remote, deps: ClaimDependen : (await repositoryInfo(client, pushed)).repository; // The README where this checkout's tree sits, which may be below the top level. const prefix = git(root, ["rev-parse", "--show-prefix"]); - let up = self; - while (up.parent !== null && (await has(client, up.parent, `${prefix}quest/README.md`))) - up = await repositoryInfo(client, up.parent); + const up = await base(client, self, prefix); return { // Reach an ancestor the way the remote is reached, so the same credentials work. git: up === self ? remote.name : /^https:/.test(source.url) ? up.https : up.ssh, @@ -85,6 +83,21 @@ export async function upstream(root: string, remote: Remote, deps: ClaimDependen }; } +type Info = Awaited>; + +/** `self`, or the nearest fork ancestor whose default branch keeps the quest tree at `prefix`. */ +async function base(client: GitHub, self: Info, prefix: string): Promise { + let up = self; + while (up.parent !== null && (await has(client, up.parent, `${prefix}quest/README.md`))) + up = await repositoryInfo(client, up.parent); + return up; +} + +/** The repository a checkout of `repository`, whose tree sits at `prefix`, opens its PRs on, as `upstream` finds it. */ +export async function baseRepository(client: GitHub, repository: Repository, prefix: string): Promise { + return (await base(client, await repositoryInfo(client, repository), prefix)).repository; +} + /** A repository as GitHub names it, its default branch and clone URLs, and its parent when it is a fork. */ async function repositoryInfo( client: GitHub, diff --git a/src/cli/gate.ts b/src/cli/gate.ts index ca6a8da2..a3f0a173 100644 --- a/src/cli/gate.ts +++ b/src/cli/gate.ts @@ -24,6 +24,7 @@ import type { Readable } from "node:stream"; import { setTimeout as sleep } from "node:timers/promises"; import { type Cancellation, cancelled, follow, forwards, GROUP, interrupts, received, requested } from "./interrupt"; import { stubbed } from "./job"; +import { envSessions, logWait } from "./waits"; import { questScope, scopeGroup, startTime, systemdScopes } from "./supervise"; /** @@ -921,9 +922,11 @@ interface Observed { export async function gate(command: string[], options: GateOptions = {}): Promise { const outermost = process.env[ADMITTED] !== "1"; const record = outermost ? recordFile(process.env) : undefined; + // An outermost gate in an agent session logs its waits for the session's record. + const waits = outermost && envSessions().length > 0; const started = Date.now(); // Read now: the command may remove its own directory. - const directory = record ? process.cwd() : ""; + const directory = record || waits ? process.cwd() : ""; const cancel = interrupts(); const observed: Observed = {}; try { @@ -936,6 +939,30 @@ export async function gate(command: string[], options: GateOptions = {}): Promis // The status the CLI exits with for a gate's own failure. status = 1; } + if (waits) { + const ended = Date.now(); + const iso = (ms: number) => new Date(ms).toISOString(); + try { + // A run cancelled before admission waited in the queue until it ended. + const admitted = observed.admitted ?? ended; + logWait( + { kind: "gate-queue", start: iso(started), end: iso(admitted), until: null }, + { cwd: directory }, + ); + if (observed.admitted !== undefined) + logWait( + { kind: "gate-run", start: iso(admitted), end: iso(ended), until: null }, + { cwd: directory }, + ); + } catch (error) { + // A cancelled run exits as it would have, logged or not. + if (cancel.signal.aborted) process.stderr.write(`quest gate: ${message(error)}\n`); + else { + failure = failure === undefined ? error : new Error(`${message(failure)}; ${message(error)}`); + status = 1; + } + } + } if (!record) { if (failure !== undefined) throw failure; return status; diff --git a/src/cli/hooks.ts b/src/cli/hooks.ts index 952f2ce2..0301e4ef 100644 --- a/src/cli/hooks.ts +++ b/src/cli/hooks.ts @@ -14,6 +14,7 @@ import { type StopInput, broken, questRoot, runningGates, usesQuest } from "./ru import { stubFindings } from "./setup"; import { CONTEXT } from "./skills"; import { HOOK_EVENTS } from "./worktree-hook"; +import { queue } from "./send"; /** The command the rule hooks run. */ export const HOOK_COMMAND = "quest hook"; @@ -44,7 +45,8 @@ const SESSION = "startup|resume|clear|compact"; * rules; SubagentStart gives each subagent the same rules, since it starts * without the session's. PreToolUse blocks the calls that break a rule a * program can check. Stop and SubagentStop refuse to end a turn while a gate - * the agent started still runs in the background. + * the agent started still runs in the background; once a session's turn ends, + * Stop queues its record to send. */ export const CLAUDE_HOOKS: readonly HookEntry[] = [ { event: "SessionStart", matcher: SESSION }, @@ -62,6 +64,7 @@ export const CLAUDE_HOOKS: readonly HookEntry[] = [ export const CODEX_HOOKS: readonly HookEntry[] = [ { event: "SessionStart", matcher: SESSION, settings: { additionalContextLimit: CONTEXT_LIMIT } }, { event: "PreToolUse", matcher: "Bash|apply_patch" }, + { event: "Stop" }, ]; /** @@ -273,6 +276,16 @@ interface HookInput { cwd: string; tool_name?: string; tool_input?: Record; + session_id?: string; + /** Null for a session that keeps no transcript. */ + transcript_path?: string | null; + /** Codex's Stop names its turn; Claude Code's does not. */ + turn_id?: string; +} + +/** What `hook` reads beyond its input. */ +export interface HookDependencies { + env: NodeJS.ProcessEnv; } /** What a Stop or SubagentStop refusal tells the agent, after the gates it names. */ @@ -288,7 +301,7 @@ const BLOCK_ON_GATES = * rule as the reason. Stop and SubagentStop refuse to end the turn while a gate * the agent started runs in the background (see `runningGates`). */ -export function hook(input: unknown): string { +export function hook(input: unknown, deps: HookDependencies = { env: process.env }): string { const value = input as Partial | null; if (typeof value?.hook_event_name !== "string" || typeof value.cwd !== "string") throw new Error("expected a Claude Code hook's JSON input with hook_event_name and cwd"); @@ -326,10 +339,17 @@ export function hook(input: unknown): string { } case "Stop": case "SubagentStop": { + // Codex runs no background gates, so its Stop only queues the record. + if ( + event === "Stop" && + typeof value.turn_id === "string" && + !Array.isArray((value as StopInput).background_tasks) + ) + return stopped(value, "codex", deps); if (!Array.isArray((value as StopInput).background_tasks)) throw new Error(`expected a ${event} hook's JSON input with background_tasks; upgrade Claude Code`); const gates = runningGates(value as StopInput); - if (gates.length === 0) return ""; + if (gates.length === 0) return event === "Stop" ? stopped(value, "claude", deps) : ""; const named = gates.map((command) => `- ${command}`).join("\n"); return JSON.stringify({ decision: "block", @@ -340,3 +360,26 @@ export function hook(input: unknown): string { throw new Error(`quest hook does not handle ${event}; run \`quest setup\` again to update your hooks`); } } + +/** + * Queue the record of the session whose turn ended for the user's sender. A Claude Code sub-agent is sent with its + * parent. Says what the person has not seen yet: why nothing is sent, that no sender runs, or a sender's failure. + */ +function stopped(value: Partial, harness: string, deps: HookDependencies): string { + // An ephemeral session, as `codex exec --ephemeral` runs, keeps no transcript, so it has no record to queue. + if (value.transcript_path === null) return ""; + if (typeof value.session_id !== "string" || typeof value.transcript_path !== "string") + throw new Error(`expected a Stop hook's JSON input with session_id and transcript_path`); + let notices: string[]; + try { + notices = queue( + { harness, session: value.session_id, transcript: value.transcript_path, cwd: value.cwd! }, + deps.env, + ).notices; + } catch (error) { + notices = [ + `quest: cannot queue this session's record: ${error instanceof Error ? error.message : String(error)}`, + ]; + } + return notices.length ? JSON.stringify({ systemMessage: notices.join("\n") }) : ""; +} diff --git a/src/cli/main.ts b/src/cli/main.ts index a02bbd68..a15bda78 100644 --- a/src/cli/main.ts +++ b/src/cli/main.ts @@ -1,5 +1,6 @@ // The `quest` command line: argument parsing and output around the core. +import { execFileSync, spawnSync } from "node:child_process"; import { existsSync, readFileSync, realpathSync } from "node:fs"; import { homedir } from "node:os"; import { join, relative, resolve, sep } from "node:path"; @@ -82,6 +83,9 @@ import { } from "./hooks"; import { CHECKED_CONTEXT, USER_ASKED, questRoot } from "./rules"; import { messageLines, messages } from "./messages"; +import { logWait, pullWaitKind, readWaits } from "./waits"; +import { CHECKOUT_CONFIG, SEND_INTERVAL, background, runSender, tokensPath } from "./send"; +import { installSender, uninstallSender } from "./service"; import { heal, init, stubFindings, uninstall } from "./setup"; import { Refusal, describe, notUpstream, releaseQuest, startQuest } from "./start"; import { CONTEXT, GUIDE, GUIDE_SECTIONS, SKILLS, guideSections, sectionHeading, skill } from "./skills"; @@ -89,7 +93,7 @@ import { type Terminal, Theme } from "./theme"; import { type Snapshot, checkout, collect, load, snapshot, under } from "./tree"; import { statusLines } from "./status"; import { pullRequestStatus } from "../github/status"; -import { TOKEN_MAX_MISSING, daysBefore, usage, usageDependencies, usageLines } from "./usage"; +import { TOKEN_MAX_MISSING, daysBefore, runTokenMax, usage, usageDependencies, usageLines } from "./usage"; import { waitForActivity, WaitTimeout } from "./wait"; import { version } from "./version"; import { Worktrees } from "./worktree"; @@ -167,7 +171,10 @@ export function main(argv: string[], out: Output): number | Promise { "With QUEST_GATE_RECORD naming an absolute path, an outermost gate on Linux or macOS appends one JSON " + "line per run to that file, however the run ends: its command, directory, when it started and was " + "admitted, its systemd scope, whose journal holds its CPU time and peak memory, and its exit status. " + - "A file it cannot write fails the gate.", + "A file it cannot write fails the gate.\n\n" + + "In an agent session, where CLAUDE_CODE_SESSION_ID or CODEX_THREAD_ID is set, an outermost gate also logs " + + "its time queued and running under $XDG_STATE_HOME/quest/waits for the session's record; a log it cannot " + + "write fails the gate.", ) .option("--quick", `Start a cheap command after at most ${QUICK_MS / 1000} seconds of waiting`) .argument("", "Executable to run") @@ -542,22 +549,37 @@ export function main(argv: string[], out: Output): number | Promise { "On timeout, the only line is a resume point for --since (a cursor, or the given --since or head SHA if setup did not finish), so a retry with it still reports activity from the first wait.\n\n" + "Uses GH_TOKEN, GITHUB_TOKEN, or gh auth token. Label changes, replies, and reactions by that account are ignored. " + "Exits 0 for activity, 3 on timeout, 2 for invalid arguments, and 1 for other failures. " + - "Issue-comment edits print as edited, including updates to older bot status comments. The account's own edits are ignored.", + "Issue-comment edits print as edited, including updates to older bot status comments. The account's own edits are ignored.\n\n" + + "In an agent session, where CLAUDE_CODE_SESSION_ID or CODEX_THREAD_ID is set, the wait is logged under " + + "$XDG_STATE_HOME/quest/waits for the session's record, named by the first activity's type, or by none on " + + "timeout; a log it cannot write exits 1 after printing the activity.", ) .argument("", "GitHub PR URL, or a PR number in origin") .option("--since ", "ISO timestamp, branch-tip SHA, or resume cursor (default: current head push time)") .option("--timeout ", "Maximum wait in seconds", positiveSeconds) .action((value: string, options: { since?: string; timeout?: number }) => { const pr = pullRequest(value, root()); + const started = new Date().toISOString(); + // An agent session's wait is logged for its record, named by the first activity, or by none on timeout. + const logged = (until: string | null): boolean => { + try { + logWait({ kind: pullWaitKind(until), start: started, end: new Date().toISOString(), until }); + return true; + } catch (error) { + warn(error instanceof Error ? error.message : String(error)); + return false; + } + }; pending = waitForActivity(pr, options, gitHubDependencies(warn)).then( (lines) => { for (const line of lines) print(line); - return 0; + return logged(lines[0]?.split(" ")[1] ?? null) ? 0 : 1; }, (error: unknown) => { if (error instanceof WaitTimeout && error.cursor) print(error.cursor); warn(error instanceof Error ? error.message : String(error)); - return error instanceof WaitTimeout ? 3 : error instanceof GitHubInputError ? 2 : 1; + if (error instanceof WaitTimeout) return logged(null) ? 3 : 1; + return error instanceof GitHubInputError ? 2 : 1; }, ); }); @@ -1148,11 +1170,19 @@ export function main(argv: string[], out: Output): number | Promise { .description( "Report origin's PRs updated since --since and the agent sessions active since then on this host and each " + "--ssh host, as token-max sessions exports and prices them, the cost an estimate at each model's API list " + - "price; its sub-agents are sessions of their own. Each Claude Code or Codex session belongs to the PR whose " + - "head branch it last names, when a repository token-max resolved for it on its own host, or for another " + - "session in the same directory there, is this one, else to the PR of the session that spawned it unless its own repository is known to be another; no prompt or reply is read out. Its role is " + - "classifying when the PR's ## Review record names it as the risk classifier, reviewing when the record names it as a reviewer or Codex marks it a review, its parent's on the same PR, implementing when its first " + - "request since --since came before the PR opened, and iterating otherwise. A session found on several hosts counts once, at its " + + "price; its sub-agents are sessions of their own. Each Claude Code or Codex session gets the record quest records " + + "sender would send for it. It belongs to the PR whose head branch it last names, when a repository token-max " + + "resolved for it on its own host, or for another session in the same directory there, is this one; a session " + + "on no such branch belongs to its parent's PRs unless its own repository is known to be another, else, as a coordinator, to every PR its sub-agents worked on, " + + "its cost counting equally toward each. Its role is classifying when a PR's ## Review record names it as the risk classifier, reviewing when it names it " + + "as a reviewer or Codex marks it a review, else the one the first quest skill it ran gives (planning, implementing, iterating, " + + "reviewing, merging, or coordinating), else its parent's unless that is coordinating, else coordinating for a session on its sub-agents' " + + "PRs only, planning on a quest-plan, quest-audit, quest-recurring, or quest-import branch, implementing when its " + + "first request since --since came before the PR opened, and iterating otherwise. Each PR sums its sessions' " + + "waits by kind: queued in and running a quest gate, waiting in quest wait for PR activity or the merge queue, " + + "each as the tool logged it for its session, and on a person's answer to a blocking question, as its " + + "transcript times it; a session on an --ssh host has no waits read. Skills and questions are read from " + + "this host's transcripts, no prompt or reply text read out. A session found on several hosts counts once, at its " + "costliest copy, and the report says how many copies it folded from which hosts; a session with a request " + "token-max could not price or a line it could not read is listed, its cost a lower bound. A host that fails fails the report. " + "Each PR also shows its ## Models size, its risk tier from the ## Review record and whether a reviewer raised it, its implementers, and as unread each Implementer line it cannot read, " + @@ -1196,6 +1226,75 @@ export function main(argv: string[], out: Output): number | Promise { ); }); + const recordsCommand = program + .command("records") + .summary("Send agent sessions' records to the Kixel Code endpoint their checkout names"); + recordsCommand + .command("sender") + .summary("Run the user's record sender until stopped, as its service does") + .description( + "Send each session record the stop hook queued, with the records of its sub-agents on this host, to the " + + `endpoint its checkout's ${CHECKOUT_CONFIG} names as records, posting {"records": [...]} with the token ` + + `${tokensPath()} pairs with that endpoint as a bearer token, one upload per endpoint and repository, ` + + `then again every ${SEND_INTERVAL / 1000} seconds until stopped, at low priority. A record is built as quest usage ` + + "builds it, from token-max's sessions on this host since the earliest queued session's first instant and " + + "the PRs updated since then, and replaces its earlier copy, so a resend never doubles. An endpoint that " + + "refuses the token (401 or 403) is sent nothing more until the token changes; any other failure keeps the " + + "sessions queued, retried after a minute and then twice as long each time up to six hours, and the next " + + "stop hook says so once. Exactly one sender runs per user, " + + "as the service quest records install sets up; start no other.\n\nUses GH_TOKEN, GITHUB_TOKEN, or gh auth token.", + ) + .action(() => { + background(); + settle( + runSender( + { + env: process.env, + github: () => gitHubDependencies(warn), + sessions: (since) => runTokenMax(since, []), + waits: (since) => readWaits(since), + fetch: globalThis.fetch, + sleep: (ms) => new Promise((done) => setTimeout(done, ms)), + now: Date.now, + interval: SEND_INTERVAL, + }, + (line) => print(`${new Date().toISOString()} ${line}`), + ), + () => 1, + ); + }); + const serviceDeps = () => ({ + platform: process.platform, + env: process.env, + uid: process.getuid?.() ?? 0, + run: (command: string, args: string[]) => { + execFileSync(command, args, { stdio: ["ignore", "inherit", "inherit"] }); + }, + tryRun: (command: string, args: string[]) => { + spawnSync(command, args, { stdio: "ignore" }); + }, + }); + recordsCommand + .command("install") + .summary("Install and start the record sender as a systemd user service or a launchd agent") + .description( + "Write the record sender's service, a systemd user service on Linux or a launchd agent on macOS running " + + "quest records sender with this shell's PATH, and (re)start it, so exactly one sender drains this user's " + + "queue. Rerun it after upgrading quest. On Linux, a user service stops at logout unless " + + "`loginctl enable-linger` keeps it; the queue holds records meanwhile.", + ) + .action(() => { + print(`installed and started ${installSender(serviceDeps())}`); + }); + recordsCommand + .command("uninstall") + .summary("Stop the record sender and remove its service") + .description("Stop the record sender and remove its service; queued records stay until a sender runs again.") + .action(() => { + const path = uninstallSender(serviceDeps()); + print(path ? `removed ${path}` : "no record sender service is installed"); + }); + program .command("messages") .summary("Print the messages a person typed to agents on this host, and their answers to agents' questions") @@ -1630,7 +1729,10 @@ export function main(argv: string[], out: Output): number | Promise { "whose body lacks a `(Written by ...)` line, or a file written under /tmp. A force-push or rebase the " + `user asked for passes with the \`${USER_ASKED}\` prefix. Stop and SubagentStop refuse to end the ` + "turn, naming each command, while a `quest gate` command the ending agent started runs in Claude " + - "Code's background: one whose background ID its transcript holds. Exits 1 on malformed input or an event it " + + "Code's background: one whose background ID its transcript holds. Otherwise Claude Code's and Codex's Stop " + + `queue the session's record for the endpoint the checkout's ${CHECKOUT_CONFIG} names, for quest records ` + + "sender to send; a checkout naming none, an endpoint the user holds no token for, or a host whose sender " + + "does not run says so once as a system message, as does a sender's failure. Exits 1 on malformed input or an event it " + "does not handle.", ) .action(() => { diff --git a/src/cli/messages.ts b/src/cli/messages.ts index a21b72c5..8e76e5d0 100644 --- a/src/cli/messages.ts +++ b/src/cli/messages.ts @@ -14,7 +14,7 @@ import { githubRepository } from "./github"; * marked incomplete when no newline ends it, as while its session is still writing it. Reading stops once `visit` * returns false. */ -function eachLine(path: string, visit: (line: string, complete: boolean) => boolean | void): void { +export function eachLine(path: string, visit: (line: string, complete: boolean) => boolean | void): void { const fd = openSync(path, "r"); try { const chunk = Buffer.alloc(1 << 22); @@ -38,7 +38,12 @@ function eachLine(path: string, visit: (line: string, complete: boolean) => bool * One JSON record of a log line, or null for an incomplete last line. With `skip`, a complete line that is not JSON, * such as a partial record a crashed write left mid-file, is counted through it and read as null instead of failing. */ -function parseLine(path: string, line: string, complete: boolean, skip?: () => void): Record | null { +export function parseLine( + path: string, + line: string, + complete: boolean, + skip?: () => void, +): Record | null { try { return JSON.parse(line) as Record; } catch (error) { @@ -200,7 +205,7 @@ function answer( const CLAUDE_SKILL = /^\s*Base directory for this skill: (?:.*\/)?([^/\n]+?)\/?[ \t]*$/m; /** The skill a Claude Code record loads, if any; built-in commands such as `/compact` load none. */ -function claudeSkill(record: Record): string | null { +export function claudeSkill(record: Record): string | null { const content = (record.message as { content?: unknown } | undefined)?.content; if (record.type !== "user" || record.isMeta !== true || !Array.isArray(content)) return null; for (const block of content) { @@ -286,10 +291,10 @@ export function readClaudeMessages( } /** Codex's tools that block on a person's answer: the first returns it, the second has it sent as a later turn. */ -const CODEX_ASK = new Set(["request_user_input", "request_user_input_async"]); +export const CODEX_ASK = new Set(["request_user_input", "request_user_input_async"]); /** The turn part carrying answers to `request_user_input_async`, as Codex's client sends them. */ -const CODEX_REPLY = /^\s*([\s\S]*)<\/send_user_message_question_reply>\s*$/; +export const CODEX_REPLY = /^\s*([\s\S]*)<\/send_user_message_question_reply>\s*$/; /** One question Codex asked: its ID for `request_user_input`'s answers, and its options. */ interface CodexQuestion { @@ -478,7 +483,7 @@ export function readCodexMessages(path: string, since: number, repository: (dir: } /** The skill a Codex `skills` turn part loads. */ -function codexSkill(text: string): string | null { +export function codexSkill(text: string): string | null { return /\s*([^<]+?)\s*<\/name>/.exec(text)?.[1] ?? null; } diff --git a/src/cli/records.ts b/src/cli/records.ts new file mode 100644 index 00000000..251ad8a1 --- /dev/null +++ b/src/cli/records.ts @@ -0,0 +1,234 @@ +// Each session's record, as `quest usage` prints it and the stop hook sends it: its PRs, role, spend, and waits. +import { type Role, type SessionRecord, type SessionWait, branchQuest } from "../core"; +import { type TranscriptFacts, transcriptFacts } from "./transcript"; +import type { Pull, Session } from "./usage"; +import type { LoggedWait } from "./waits"; + +/** The role the first quest skill a session ran gives it; a skill not listed gives none. */ +const SKILL_ROLES: Record = { + "quest-plan": "planning", + "quest-audit": "planning", + "quest-recurring": "planning", + "quest-import": "planning", + "quest-grill": "planning", + "quest-delete": "planning", + "quest-export": "planning", + "quest-start": "implementing", + "quest-iterate": "iterating", + "quest-review": "reviewing", + "quest-merge": "merging", + "quest-complete": "merging", + "quest-run": "coordinating", + "quest-spawn": "coordinating", +}; + +/** Branch prefixes of the skills whose PRs plan quests. */ +const PLANNING = ["quest-plan/", "quest-audit/", "quest-recurring/", "quest-import/"]; + +/** What the records are built from. */ +export interface RecordInput { + /** The base repository as lowercase `owner/repo`. */ + repository: string; + /** The sessions to build records for, each once. */ + sessions: Session[]; + /** The PR each session's own branch heads, from `attribute`. */ + own: Map; + /** The repositories a session is known to be in, or null when unknown. */ + known?: (session: Session) => string[] | null; + /** Session IDs a PR's review record names as reviewers, and as its risk classifier. */ + reviewers: Set; + classifiers: Set; + /** The waits logged on this host, which only its own sessions (host null) are given. */ + waits: LoggedWait[]; + /** The sessions to build records for; by default, those on a PR. Every session still counts for lineage. */ + emit?: Set; + /** The window's start, epoch ms: a person's wait that ended before it, like a logged one, is left out. */ + since: number; +} + +const id = (harness: string, session: string) => `${harness}\0${session}`; +const iso = (ms: number) => new Date(ms).toISOString(); + +/** + * A record for each session that `emit` names, or that worked on a PR. A session's PRs are its own, else its parent's, else, for one that spawned + * sub-agents, every PR they worked on. Its role is classifying when a PR's review record names it as the risk + * classifier, reviewing when Codex marks it a review or a PR's review record + * names it, else the one its first quest skill gives, else its parent's unless that is coordinating, else coordinating for a + * session whose PRs are only its sub-agents', planning on a planning skill's branch, and implementing or iterating + * by whether it started before its PR opened. Transcripts and waits are read only for this host's sessions. + */ +export function buildRecords(input: RecordInput): Map { + const named = input.sessions.filter((s): s is Session & { id: string } => s.id !== null); + const byId = new Map(named.map((s) => [id(s.harness, s.id), s])); + const children = new Map(); + for (const s of input.sessions) { + const parent = s.parent === null ? undefined : byId.get(id(s.harness, s.parent)); + if (parent && parent !== s) children.set(parent, [...(children.get(parent) ?? []), s]); + } + const parentOf = (s: Session) => (s.parent === null ? undefined : byId.get(id(s.harness, s.parent))); + + const descendants = (s: Session, seen = new Set()): Pull[] => { + seen.add(s); + return (children.get(s) ?? []).flatMap((child) => { + if (seen.has(child)) return []; + const own = input.own.get(child); + return own ? [own, ...descendants(child, seen)] : descendants(child, seen); + }); + }; + // Bounded by the sessions' count, so a cycle in recorded lineage cannot loop. + const pullsOf = (s: Session, depth = 0): { pulls: Pull[]; via: "own" | "parent" | "children" | "none" } => { + const own = input.own.get(s); + if (own) return { pulls: [own], via: "own" }; + const parent = parentOf(s); + if (parent && depth < input.sessions.length) { + // A session known to be in another repository keeps out of its parent's PRs, as a sub-agent in another + // checkout does. + const known = input.known?.(s) ?? null; + const inherited = pullsOf(parent, depth + 1).pulls.filter( + (p) => known === null || (p.repository !== null && known.includes(p.repository)), + ); + if (inherited.length) return { pulls: inherited, via: "parent" }; + } + const found = descendants(s); + const unique = [...new Map(found.map((p) => [p.number, p])).values()].sort((a, b) => a.number - b.number); + return unique.length ? { pulls: unique, via: "children" } : { pulls: [], via: "none" }; + }; + + const facts = new Map(); + const factsOf = (s: Session): TranscriptFacts | null => { + if (s.host !== null) return null; + if (!facts.has(s)) facts.set(s, readFacts(s)); + return facts.get(s)!; + }; + + const roleOf = (s: Session, depth = 0): Role => { + if (s.id !== null && input.classifiers.has(s.id)) return "classifying"; + if (s.review || (s.id !== null && input.reviewers.has(s.id))) return "reviewing"; + const skill = factsOf(s)?.skill; + const fromSkill = skill ? SKILL_ROLES[skill] : undefined; + if (fromSkill) return fromSkill; + const own = input.own.get(s); + const parent = parentOf(s); + // A sub-agent with no skill of its own works for its parent, unless that parent hands out whole PRs. + if (parent && depth < input.sessions.length) { + const inherited = roleOf(parent, depth + 1); + if (inherited !== "coordinating") return inherited; + } + const branch = own?.branch ?? lastBranch(s); + if (branch && PLANNING.some((prefix) => branch.startsWith(prefix))) return "planning"; + if (!own) return pullsOf(s).via === "children" ? "coordinating" : "implementing"; + return s.start < own.created ? "implementing" : "iterating"; + }; + + const waits = assignWaits(named, input.waits, parentOf, input.since); + const records = new Map(); + for (const s of input.sessions) { + const { pulls, via } = pullsOf(s); + if (input.emit ? !input.emit.has(s) : pulls.length === 0) continue; + const own = input.own.get(s); + const branch = own?.branch ?? (via === "none" ? lastBranch(s) : pulls.length === 1 ? pulls[0]!.branch : null); + const local = factsOf(s); + const person: SessionWait[] = (local?.questions ?? []) + .filter((q) => q.end >= input.since) + .map((q) => ({ + kind: "person", + start: iso(q.start), + end: iso(q.end), + until: null, + })); + records.set(s, { + harness: s.harness, + session: s.id, + parent: s.parent, + repository: input.repository, + branch, + quest: branch?.startsWith("quest/") ? branchQuest(branch) : null, + pulls: pulls.map((p) => p.number), + role: roleOf(s), + models: s.models.map((m) => ({ + model: m.model, + effort: m.effort, + input: m.input, + cacheWrite: m.cacheWrite, + cacheRead: m.cacheRead, + output: m.output, + cost: m.cost, + complete: m.complete, + })), + cost: s.cost, + complete: s.complete, + start: iso(s.start), + end: iso(s.end), + waits: + local === null + ? null + : [ + ...(waits.get(s) ?? []).map(({ kind, start, end, until }) => ({ kind, start, end, until })), + ...person, + ].sort((a, b) => a.start.localeCompare(b.start)), + }); + } + return records; +} + +/** + * A local session's transcript facts, or null when they cannot be read: a file deleted since token-max read it, or + * one with lines token-max could not read either, whose cost is already a lower bound. Any other failure is a shape + * quest does not know, and throws. + */ +function readFacts(s: Session): TranscriptFacts | null { + try { + return transcriptFacts(s.harness, s.files); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT" || s.skippedLines > 0) return null; + throw error; + } +} + +/** The branch a session named last, or null. */ +function lastBranch(s: Session): string | null { + return [...s.branches].sort((a, b) => b.last - a.last)[0]?.name ?? null; +} + +/** + * Each logged wait's session on this host. A wait names the sessions in its command's environment; where it names + * several, as when one harness started the other, the innermost is the one that started last. A Claude sub-agent's + * commands carry its parent's ID, so a wait goes to the one sub-agent of it running then in the wait's directory, + * the deepest checkout matching, and stays with the parent when none or several match. + */ +function assignWaits( + sessions: (Session & { id: string })[], + logged: LoggedWait[], + parentOf: (s: Session) => Session | undefined, + since: number, +): Map { + const local = sessions.filter((s) => s.host === null); + const byId = new Map(local.map((s) => [id(s.harness, s.id), s])); + const assigned = new Map(); + for (const wait of logged) { + const start = Date.parse(wait.start); + const end = Date.parse(wait.end); + // A session's start is its first request in the window, so before the window no start orders a wait: any + // session it names may own it, and a sub-agent owns it by running at any point of it. + const clipped = start < since; + const named = wait.sessions.flatMap((n) => byId.get(id(n.harness, n.session)) ?? []); + const owner = named.filter((s) => clipped || s.start <= start).sort((a, b) => b.start - a.start)[0]; + if (!owner) continue; + const inside = (dir: string | null) => + dir !== null && (wait.directory === dir || wait.directory.startsWith(dir.endsWith("/") ? dir : `${dir}/`)); + const running = local.filter( + (s) => + s.harness === "claude" && + parentOf(s) === owner && + (clipped ? s.start <= end : s.start <= start) && + start <= s.end && + inside(s.cwd), + ); + const depth = Math.max(-1, ...running.map((s) => s.cwd!.length)); + const deepest = running.filter((s) => s.cwd!.length === depth); + // A parent waits on its foreground sub-agent, so the one sub-agent running there is the waiter. + const target = deepest.length === 1 ? deepest[0]! : owner; + assigned.set(target, [...(assigned.get(target) ?? []), wait]); + } + return assigned; +} diff --git a/src/cli/review-record.ts b/src/cli/review-record.ts index 86e02408..d45800ad 100644 --- a/src/cli/review-record.ts +++ b/src/cli/review-record.ts @@ -159,6 +159,30 @@ export interface ReviewSummary { const MARK = "Rewritten by `quest review` after each round."; +/** + * The reviewer sessions `body`'s `## Review` record names, read from its `Session:` lines alone, so a record whose + * other lines this version cannot parse, as one a newer `quest review` wrote, still names them. + */ +export function reviewSessions(body: string): string[] { + const section = reviewSection(body); + if (!section) return []; + const all = body.slice(section.start, section.end); + const at = all.indexOf(MARK); + if (at < 0) return []; + return [...all.slice(at).matchAll(/^- Session: (`+) ?(.*?) ?\1(?:, round \d+)?\r?$/gm)].map((found) => found[2]!); +} + +/** + * The risk classifier session `body`'s `## Review` record names, read from its `Classified:` line alone, as + * `reviewSessions` reads reviewers. + */ +export function classifierSessions(body: string): string[] { + const section = reviewSection(body); + if (!section) return []; + const found = /^- Classified: \S+ by .*, session (`+) ?(.*?) ?\1\r?$/m.exec(body.slice(section.start, section.end)); + return found ? [found[2]!] : []; +} + /** * Each reviewer's latest round `body`'s `## Review` section records, or none when `quest review` wrote none * there, as when the section records a skipped review or only the land fixer's fixes. diff --git a/src/cli/send.ts b/src/cli/send.ts new file mode 100644 index 00000000..f65fdfff --- /dev/null +++ b/src/cli/send.ts @@ -0,0 +1,534 @@ +// Sending each session's record to the Kixel Code endpoint its checkout names. The stop hook only queues the session; +// one sender per user, run by its service manager, rebuilds each queued session's record, with its sub-agents', and +// posts them, keeping what failed for the next pass. A record replaces its earlier copy, so a resend never doubles. +import { spawnSync } from "node:child_process"; +import { existsSync, mkdirSync, readdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; +import { homedir, setPriority } from "node:os"; +import { dirname, join } from "node:path"; +import { createHash, randomUUID } from "node:crypto"; + +import { type SessionRecord, parseUpload } from "../core"; +import { type GitHubDependencies, GitHub } from "../github/client"; +import { baseRepository } from "./claims"; +import { discoveryEnv } from "./git"; +import { readRemote, remoteRepository } from "./github"; +import { eachLine } from "./messages"; +import { questRoot } from "./rules"; +import { type Pull, type Session, fold, key, sessionRecords, updatedPulls } from "./usage"; +import { type LoggedWait } from "./waits"; + +/** The checkout's file naming its record endpoint, as `{"records": "https://..."}`. */ +export const CHECKOUT_CONFIG = ".quest.json"; + +/** One session the stop hook queued. */ +export interface Pending { + harness: string; + session: string; + /** Its transcript's first instant, epoch ms, so token-max reads all of it. */ + start: number; + /** A fresh ID per stop, naming its own queue file, so a sender removes only the stops it sent. */ + queued: string; + /** The checkout's quest root, its `origin` on GitHub as `owner/repo`, and where its quest tree sits below the top level. */ + root: string; + origin: string; + prefix: string; + endpoint: string; +} + +/** Where the sender keeps its queue and what it has said. */ +export function stateDirectory(env: NodeJS.ProcessEnv): string { + return join(env.XDG_STATE_HOME || join(homedir(), ".local", "state"), "quest", "records"); +} + +/** The user's tokens, by the endpoint each may be sent to: `{"https://...": "token"}`. */ +export function tokensPath(env: NodeJS.ProcessEnv = process.env): string { + return join(env.XDG_CONFIG_HOME || join(homedir(), ".config"), "quest", "records.json"); +} + +/** The endpoint `root`'s checkout names, or null; a malformed file or endpoint throws. */ +export function checkoutEndpoint(root: string): string | null { + const path = join(root, CHECKOUT_CONFIG); + if (!existsSync(path)) return null; + let config: unknown; + try { + config = JSON.parse(readFileSync(path, "utf8")); + } catch (error) { + throw new Error(`${path} is not JSON: ${(error as Error).message}`); + } + const endpoint = (config as { records?: unknown } | null)?.records; + if (endpoint === undefined) return null; + if (typeof endpoint !== "string") throw new Error(`${path}: records is not a URL`); + return checkedEndpoint(endpoint, path); +} + +/** `endpoint` when it is HTTPS, or HTTP to this host for a local store; anything else would send records in clear. */ +function checkedEndpoint(endpoint: string, where: string): string { + let url: URL; + try { + url = new URL(endpoint); + } catch { + throw new Error(`${where}: ${endpoint} is not a URL`); + } + const loopback = ["localhost", "127.0.0.1", "[::1]"].includes(url.hostname); + if (url.protocol !== "https:" && !(url.protocol === "http:" && loopback)) + throw new Error(`${where}: ${endpoint} is not an HTTPS URL`); + return endpoint; +} + +/** + * The token the user paired with `endpoint`, or null. The token is bound to its endpoint in the user's own file, so + * a checkout naming another endpoint, as a hostile branch might, never receives it. + */ +export function tokenFor(endpoint: string, env: NodeJS.ProcessEnv = process.env): string | null { + const path = tokensPath(env); + if (!existsSync(path)) return null; + let tokens: unknown; + try { + tokens = JSON.parse(readFileSync(path, "utf8")); + } catch (error) { + throw new Error(`${path} is not JSON: ${(error as Error).message}`); + } + if (typeof tokens !== "object" || tokens === null || Array.isArray(tokens)) + throw new Error(`${path} does not hold an object of tokens by endpoint`); + const token = (tokens as Record)[endpoint]; + if (token === undefined) return null; + if (typeof token !== "string" || token === "") + throw new Error(`${path}: the token for ${endpoint} is not a string`); + return token; +} + +/** The marker of an endpoint that refused a token, which a new token clears. */ +const refusal = (env: NodeJS.ProcessEnv, endpoint: string, token: string) => + join(stateDirectory(env), "refused", digest(`${endpoint}\0${token}`)); + +function refuse(env: NodeJS.ProcessEnv, endpoint: string, token: string): void { + mkdirSync(join(stateDirectory(env), "refused"), { recursive: true, mode: 0o700 }); + writeFileSync(refusal(env, endpoint, token), ""); +} + +/** Say `notice` the first time only, by `key`, returning it then and null after. */ +function once(env: NodeJS.ProcessEnv, key: string, notice: string): string | null { + const dir = join(stateDirectory(env), "said"); + const marker = join(dir, digest(key)); + if (existsSync(marker)) return null; + mkdirSync(dir, { recursive: true, mode: 0o700 }); + writeFileSync(marker, `${key}\n`); + return notice; +} + +const digest = (text: string) => createHash("sha256").update(text).digest("hex").slice(0, 32); + +/** The first instant a transcript records, epoch ms, or null when it records none. */ +function transcriptStart(path: string): number | null { + let found: number | null = null; + eachLine(path, (line) => { + const at = /"timestamp"\s*:\s*"([^"]+)"/.exec(line)?.[1]; + const ms = at === undefined ? NaN : Date.parse(at); + if (Number.isNaN(ms)) return true; + found = ms; + return false; + }); + return found; +} + +/** A stop hook's input, as Claude Code and Codex send it. */ +export interface Stopped { + harness: string; + session: string; + transcript: string; + cwd: string; +} + +/** What `queue` did, for the hook to say. */ +export interface Queued { + /** Each notice the person has not seen yet. */ + notices: string[]; + /** Whether the session was queued. */ + queued: boolean; +} + +/** + * Queue the stopped session's record for its checkout's endpoint, in a checkout that uses Quest. A checkout naming + * no endpoint, or one the user holds no token for, queues nothing and says so once. A sender's last failure, if the + * person has not seen it, is said too. + */ +export function queue(stopped: Stopped, env: NodeJS.ProcessEnv = process.env): Queued { + const notices: string[] = []; + const failure = unseenFailure(env); + if (failure) notices.push(failure); + const root = questRoot(stopped.cwd); + if (root === null) return { notices, queued: false }; + const said = (key: string, notice: string) => { + const text = once(env, key, notice); + if (text) notices.push(text); + return { notices, queued: false }; + }; + const run = (args: string[]) => spawnSync("git", ["-C", root, ...args], { encoding: "utf8", env: discoveryEnv() }); + const remote = readRemote(root, "origin"); + const origin = remote === null ? null : remoteRepository(remote).repository; + // A checkout off GitHub has no PRs for a record to name. + if (origin === null) return { notices, queued: false }; + const repository = `${origin.owner}/${origin.repo}`.toLowerCase(); + const endpoint = checkoutEndpoint(root); + if (endpoint === null) + return said( + `none:${repository}`, + `quest: ${repository} names no Kixel Code endpoint in ${CHECKOUT_CONFIG}, so its sessions' records stay on this host.`, + ); + const token = tokenFor(endpoint, env); + if (token === null) + return said( + `token:${endpoint}`, + `quest: ${tokensPath(env)} holds no token for ${endpoint}, so this host sends it no session records.`, + ); + if (existsSync(refusal(env, endpoint, token))) return { notices, queued: false }; + const dir = join(stateDirectory(env), "pending"); + const start = transcriptStart(stopped.transcript); + if (start === null) throw new Error(`${stopped.transcript} records no instant, so its session cannot be read`); + const queued = randomUUID(); + const pending: Pending = { + harness: stopped.harness, + session: stopped.session, + start, + queued, + root, + origin: repository, + prefix: run(["rev-parse", "--show-prefix"]).stdout.trim(), + endpoint, + }; + mkdirSync(dir, { recursive: true, mode: 0o700 }); + // Each stop gets a file of its own, never rewritten, so a sender removes only the stops it read. Written whole, + // then renamed, so the sender never reads half a file. + const path = join(dir, `${queued}.json`); + writeFileSync(`${path}.new`, JSON.stringify(pending)); + renameSync(`${path}.new`, path); + const sender = senderRuns(env); + if (!sender.runs) + // Said once per outage: the PID of the sender that stopped names it, or none for a host that never ran one. + said( + `down:${sender.pid}`, + `quest: no session record sender runs on this host, so this session's record waits in ${dir} with any others; start it with \`quest records install\`.`, + ); + return { notices, queued: true }; +} + +/** The sender's last failure when the person has not seen it yet, marking it seen. */ +function unseenFailure(env: NodeJS.ProcessEnv): string | null { + const path = join(stateDirectory(env), "failure"); + if (!existsSync(path)) return null; + const text = readFileSync(path, "utf8"); + const seen = `${path}.seen`; + if (existsSync(seen) && readFileSync(seen, "utf8") === text) return null; + writeFileSync(seen, text); + return `quest: ${text.trim()}`; +} + +function writeFailure(env: NodeJS.ProcessEnv, text: string): void { + const dir = stateDirectory(env); + mkdirSync(dir, { recursive: true, mode: 0o700 }); + writeFileSync(join(dir, "failure"), `${text}\n`); +} + +/** What the sender reads and posts with. */ +export interface SendDependencies { + env: NodeJS.ProcessEnv; + github(): GitHubDependencies; + /** token-max's sessions on this host since an instant, or null when it is not installed. */ + sessions(since: Date): Session[] | null; + waits(since: number): LoggedWait[]; + fetch: typeof globalThis.fetch; + now(): number; + sleep(ms: number): Promise; + /** The time between passes, and a failing group's first wait. */ + interval: number; +} + +/** How long the sender waits between passes, so a busy host's stops cannot spend its GitHub quota. */ +export const SEND_INTERVAL = 60_000; + +/** The outcome of one upload. */ +export interface Sent { + endpoint: string; + repository: string; + records: number; + /** The endpoint refused the token, so the records were dropped. */ + refused: boolean; +} + +/** Where the running sender records its process ID, for the stop hook to check. */ +const pidPath = (env: NodeJS.ProcessEnv) => join(stateDirectory(env), "sender.pid"); + +/** + * Whether a sender runs: the process the sender's PID file names is alive. A reused PID reads as running, which only + * delays the notice; the queue keeps every record meanwhile. + */ +function senderRuns(env: NodeJS.ProcessEnv): { runs: boolean; pid: string } { + const path = pidPath(env); + const pid = existsSync(path) ? readFileSync(path, "utf8").trim() : ""; + const owner = Number(pid); + return { runs: Number.isSafeInteger(owner) && owner > 0 && alive(owner), pid }; +} + +function alive(pid: number): boolean { + try { + process.kill(pid, 0); + return true; + } catch (error) { + return (error as NodeJS.ErrnoException).code === "EPERM"; + } +} + +/** + * Run the sender until stopped: each `interval`, send every queued session's record, keeping what failed for the + * next pass and leaving the failure for the stop hook to say. The user's service manager runs exactly one, so no two + * passes ever overlap and uploads of a session reach the store in order. + */ +export async function runSender(deps: SendDependencies, log: (line: string) => void): Promise { + const dir = stateDirectory(deps.env); + mkdirSync(join(dir, "pending"), { recursive: true, mode: 0o700 }); + writeFileSync(pidPath(deps.env), String(process.pid)); + for (;;) { + try { + for (const each of await sendPass(deps)) + log( + each.refused + ? `${each.endpoint} refused the token for ${each.repository}; ${each.records} records dropped` + : `sent ${each.records} records of ${each.repository} to ${each.endpoint}`, + ); + } catch (error) { + log(error instanceof Error ? error.message : String(error)); + } + await deps.sleep(deps.interval); + } +} + +/** + * Send every queued session's record once, with those of its sub-agents on this host, one upload per endpoint and + * repository. A record goes only to the endpoint its token is paired with. An upload refused for access drops its + * sessions and says so once; any other failure keeps them queued for the next pass, records it for the stop hook, + * and throws. A session queued again while an upload ran stays queued. + */ +export async function sendPass(deps: SendDependencies): Promise { + const dir = stateDirectory(deps.env); + try { + return await sendOnce(deps, dir); + } catch (error) { + writeFailure( + deps.env, + `session records were not sent and stay queued, retried with backoff: ${error instanceof Error ? error.message : String(error)}`, + ); + throw error; + } +} + +async function sendOnce(deps: SendDependencies, dir: string): Promise { + const queued = readdirSync(join(dir, "pending")) + .filter((name) => name.endsWith(".json")) + .map((name) => { + const path = join(dir, "pending", name); + return { path, pending: JSON.parse(readFileSync(path, "utf8")) as Pending }; + }); + const groups = new Map(); + for (const q of queued) { + const id = JSON.stringify([q.pending.endpoint, q.pending.origin, q.pending.prefix]); + groups.set(id, [...(groups.get(id) ?? []), q]); + } + // A group backing off sits the pass out, so it neither reruns token-max nor widens the others' window. + for (const id of [...groups.keys()]) if ((readBackoff(deps.env, id)?.next ?? 0) > deps.now()) groups.delete(id); + const active = [...groups.values()].flat(); + if (active.length === 0) return []; + const since = Math.min(...active.map((q) => q.pending.start)); + const exported = deps.sessions(new Date(since)); + if (exported === null) + throw new Error( + "token-max is not installed, so no record can be priced; install it with `cargo install --locked --git https://github.com/kixel-ai/token-max`", + ); + const { kept } = fold(exported); + const byId = new Map(kept.flatMap((s) => (s.id === null ? [] : [[`${s.harness}\0${s.id}`, s] as const]))); + const waits = deps.waits(since); + const client = new GitHub(deps.github()); + const sent: Sent[] = []; + const failures: string[] = []; + const notes: string[] = []; + for (const [id, group] of groups) { + const { endpoint, origin, prefix } = group[0]!.pending; + const roots = new Set(group.map((q) => q.pending.root)); + try { + const token = tokenFor(endpoint, deps.env); + // The user removed the token since the session was queued, or the endpoint refused it: it sends nothing. + if (token === null || existsSync(refusal(deps.env, endpoint, token))) { + for (const q of group) settle(q); + continue; + } + const [owner, repo] = origin.split("/") as [string, string]; + const repository = await baseRepository(client, { owner, repo }, prefix); + const pulls: Pull[] = await updatedPulls(client, repository, since); + const emit = new Set(); + const missing: typeof group = []; + for (const q of group) { + const session = byId.get(`${q.pending.harness}\0${q.pending.session}`); + if (session) emit.add(session); + else missing.push(q); + } + // A session token-max no longer lists, as when its transcript was deleted, can never be built: it is + // dropped and said, so it does not hold back the rest of its group forever. + for (const q of missing) { + settle(q); + notes.push( + `token-max lists no ${q.pending.harness} session ${q.pending.session} on this host, so its record was dropped`, + ); + } + const repositories = new Set([origin, key(repository)]); + // A sub-agent stops without the stop hook naming it, so each queued session brings its own, routed as its + // own checkout routes it, found from the nearest folder still there: a checkout must be a queued one's, or + // name the same origin and endpoint. With no checkout left, every repository token-max resolved for it + // must be this group's. + const routed = (s: Session): boolean => { + // Repositories token-max resolved that are none of this group's, as for a deleted nested clone of + // another, rule it out whatever checkout now holds its folder. + if ( + s.repositories !== null && + s.repositories.length > 0 && + !s.repositories.some((r) => repositories.has(r)) + ) + return false; + if (s.cwd !== null && insideCheckout(s.cwd)) { + const own = questRoot(s.cwd); + if (own === null) return false; + if (roots.has(own)) return true; + const remote = readRemote(own, "origin"); + const named = remote === null ? null : remoteRepository(remote).repository; + try { + return named !== null && key(named) === origin && checkoutEndpoint(own) === endpoint; + } catch { + // A malformed .quest.json routes nowhere, as its own stop would say. + return false; + } + } + return ( + s.repositories !== null && + s.repositories.length > 0 && + s.repositories.every((r) => repositories.has(r)) + ); + }; + for (let grew = true; grew;) { + grew = false; + for (const s of kept) { + const parent = s.parent === null ? undefined : byId.get(`${s.harness}\0${s.parent}`); + if (s.id !== null && !emit.has(s) && parent !== undefined && emit.has(parent) && routed(s)) { + emit.add(s); + grew = true; + } + } + } + if (emit.size === 0) continue; + const records = sessionRecords({ + repository: key(repository), + sessions: kept, + evidence: exported, + pulls, + waits, + emit, + since, + }); + const upload = [...records.values()]; + try { + parseUpload({ records: upload }); + } catch (error) { + throw new Error(`this quest built an upload its own parseUpload refuses: ${(error as Error).message}`); + } + const response = await deps.fetch(endpoint, { + method: "POST", + headers: { authorization: `Bearer ${token}`, "content-type": "application/json" }, + body: JSON.stringify({ records: upload } satisfies { records: SessionRecord[] }), + }); + if (response.status === 401 || response.status === 403) { + refuse(deps.env, endpoint, token); + clearBackoff(deps.env, id); + for (const q of group) settle(q); + notes.push( + `${endpoint} refused this host's token (${response.status}), so it is sent no more session records until the token changes`, + ); + sent.push({ endpoint, repository: key(repository), records: upload.length, refused: true }); + continue; + } + if (!response.ok) + throw new Error(`${endpoint} answered ${response.status}: ${(await response.text()).slice(0, 500)}`); + clearBackoff(deps.env, id); + for (const q of group) settle(q); + sent.push({ endpoint, repository: key(repository), records: upload.length, refused: false }); + } catch (error) { + const wait = backOff(deps, id); + failures.push( + `${origin} to ${endpoint}: ${error instanceof Error ? error.message : String(error)} (tried again in ${Math.round(wait / 60_000)} minutes)`, + ); + } + } + if (failures.length) throw new Error([...notes, ...failures].join("; ")); + if (notes.length) writeFailure(deps.env, notes.join("; ")); + else rmSync(join(dir, "failure"), { force: true }); + return sent; +} + +/** The longest a failing group waits between tries. */ +const MAX_BACKOFF = 6 * 3_600_000; + +/** A failing group's consecutive failures and the instant it may next be tried. */ +interface Backoff { + failures: number; + next: number; +} + +const backoffPath = (env: NodeJS.ProcessEnv, id: string) => join(stateDirectory(env), "backoff", digest(id)); + +/** A group's backoff, or null without one; an unreadable file reads as none, so it can never block every group. */ +function readBackoff(env: NodeJS.ProcessEnv, id: string): Backoff | null { + try { + const value = JSON.parse(readFileSync(backoffPath(env, id), "utf8")) as Partial | null; + return Number.isSafeInteger(value?.failures) && Number.isFinite(value?.next) ? (value as Backoff) : null; + } catch { + return null; + } +} + +/** + * Record another failure of group `id`, doubling its wait from one interval up to `MAX_BACKOFF`, so a rejection that + * resending cannot fix, such as a store refusing a newer record, costs a pass a few times a day while its records + * stay queued. Returns the wait. + */ +function backOff(deps: SendDependencies, id: string): number { + const failures = (readBackoff(deps.env, id)?.failures ?? 0) + 1; + const wait = Math.min(deps.interval * 2 ** Math.min(failures - 1, 30), MAX_BACKOFF); + mkdirSync(join(stateDirectory(deps.env), "backoff"), { recursive: true, mode: 0o700 }); + // Written whole, then renamed, as queue files are, so a crash mid-write leaves the old file or none. + const path = backoffPath(deps.env, id); + writeFileSync(`${path}.new`, JSON.stringify({ failures, next: deps.now() + wait } satisfies Backoff)); + renameSync(`${path}.new`, path); + return wait; +} + +function clearBackoff(env: NodeJS.ProcessEnv, id: string): void { + rmSync(backoffPath(env, id), { force: true }); +} + +/** Whether a checkout still holds `dir` or the nearest of its parents that exists, as `questRoot` finds it. */ +function insideCheckout(dir: string): boolean { + for (let at = dir; ; at = dirname(at)) { + if (existsSync(join(at, ".git"))) return true; + if (dirname(at) === at) return false; + } +} + +/** Drop a stop the sender read; a later stop of the same session has a file of its own and stays queued. */ +function settle(q: { path: string }): void { + rmSync(q.path, { force: true }); +} + +/** Lower this process's priority, since the sender works in the background. */ +export function background(): void { + try { + setPriority(19); + } catch { + // A host that refuses only leaves it at normal priority. + } +} diff --git a/src/cli/service.ts b/src/cli/service.ts new file mode 100644 index 00000000..84d59540 --- /dev/null +++ b/src/cli/service.ts @@ -0,0 +1,144 @@ +// Installing the per-user record sender as a systemd user service or a launchd agent, so the service manager runs +// exactly one and restarts it. +import { accessSync, constants, existsSync, mkdirSync, rmSync, writeFileSync } from "node:fs"; +import { homedir } from "node:os"; +import { delimiter, isAbsolute, join } from "node:path"; + +import { stateDirectory } from "./send"; + +/** The systemd unit and launchd label the sender runs as. */ +const UNIT = "quest-records.service"; +const LABEL = "dev.moq.quest.records"; + +/** What installing the sender reads and runs. */ +export interface ServiceDependencies { + platform: NodeJS.Platform; + env: NodeJS.ProcessEnv; + /** The user's ID, for launchd's per-user domain. */ + uid: number; + /** Run a service manager command, throwing when it fails. */ + run(command: string, args: string[]): void; + /** Run one whose failure is expected, such as stopping a service that is not loaded. */ + tryRun(command: string, args: string[]): void; +} + +const home = (env: NodeJS.ProcessEnv) => env.HOME || homedir(); + +function supported(platform: NodeJS.Platform): "linux" | "darwin" { + if (platform === "linux" || platform === "darwin") return platform; + throw new Error(`the record sender runs as a systemd user service or a launchd agent, which ${platform} lacks`); +} + +/** Where the service's definition lives. */ +export function servicePath(platform: NodeJS.Platform, env: NodeJS.ProcessEnv): string { + return supported(platform) === "linux" + ? join(env.XDG_CONFIG_HOME || join(home(env), ".config"), "systemd", "user", UNIT) + : join(home(env), "Library", "LaunchAgents", `${LABEL}.plist`); +} + +/** `quest` as PATH finds it, unresolved, so an upgrade that moves the target behind a profile link is followed. */ +function questOnPath(env: NodeJS.ProcessEnv): string { + for (const dir of (env.PATH ?? "").split(delimiter)) { + if (!isAbsolute(dir)) continue; + const path = join(dir, "quest"); + try { + accessSync(path, constants.X_OK); + return path; + } catch { + // Not here. + } + } + throw new Error("quest is not on PATH, so no service can run it"); +} + +const xml = (text: string) => + text.replace(/&/g, "&").replace(//g, ">").replace(/"/g, """); + +/** systemd's quoting for one word, its specifiers and, for ExecStart, its variables escaped. */ +const systemdWord = (text: string) => `"${text.replace(/\\/g, "\\\\").replace(/"/g, '\\"').replace(/%/g, "%%")}"`; +const execWord = (text: string) => systemdWord(text).replace(/\$/g, "$$$$"); + +/** + * What the sender reads beyond PATH, kept from the installer's environment, since the service manager's may lack + * them: where the hooks queue and the tokens live, where token-max finds transcripts, and gh's login. Secrets such + * as GH_TOKEN are never written into the definition. + */ +const KEPT = ["XDG_CONFIG_HOME", "XDG_STATE_HOME", "XDG_DATA_HOME", "CLAUDE_CONFIG_DIR", "CODEX_HOME", "GH_CONFIG_DIR"]; + +/** + * The service definition running `quest records sender` with the installer's PATH, so it finds token-max, gh, and the + * runtime `quest` needs, as the hooks do, and the directories `KEPT` names. + */ +export function serviceDefinition(platform: NodeJS.Platform, env: NodeJS.ProcessEnv): string { + const kind = supported(platform); + const quest = questOnPath(env); + const vars: [string, string][] = [["PATH", env.PATH ?? ""]]; + for (const name of KEPT) if (env[name]) vars.push([name, env[name]!]); + if (kind === "linux") + return [ + "[Unit]", + "Description=Send agent session records (quest records sender)", + "", + "[Service]", + `ExecStart=${execWord(quest)} records sender`, + ...vars.map(([name, value]) => `Environment=${systemdWord(`${name}=${value}`)}`), + "Restart=always", + "RestartSec=60", + "", + "[Install]", + "WantedBy=default.target", + "", + ].join("\n"); + const log = join(stateDirectory(env), "sender.log"); + return ` + + + + Label${LABEL} + ProgramArguments + ${xml(quest)}recordssender + EnvironmentVariables + ${vars.map(([name, value]) => `${name}${xml(value)}`).join("")} + RunAtLoad + KeepAlive + ThrottleInterval60 + ProcessTypeBackground + StandardOutPath${xml(log)} + StandardErrorPath${xml(log)} + + +`; +} + +/** Write the sender's service and (re)start it, so a reinstall after an upgrade runs the new `quest`. */ +export function installSender(deps: ServiceDependencies): string { + const path = servicePath(deps.platform, deps.env); + const definition = serviceDefinition(deps.platform, deps.env); + mkdirSync(join(path, ".."), { recursive: true }); + mkdirSync(stateDirectory(deps.env), { recursive: true, mode: 0o700 }); + writeFileSync(path, definition); + if (deps.platform === "linux") { + deps.run("systemctl", ["--user", "daemon-reload"]); + deps.run("systemctl", ["--user", "enable", UNIT]); + deps.run("systemctl", ["--user", "restart", UNIT]); + } else { + deps.tryRun("launchctl", ["bootout", `gui/${deps.uid}/${LABEL}`]); + deps.run("launchctl", ["bootstrap", `gui/${deps.uid}`, path]); + } + return path; +} + +/** Stop the sender and remove its service; queued records stay until a sender runs again. */ +export function uninstallSender(deps: ServiceDependencies): string | null { + const path = servicePath(deps.platform, deps.env); + if (!existsSync(path)) return null; + if (deps.platform === "linux") { + deps.run("systemctl", ["--user", "disable", "--now", UNIT]); + rmSync(path); + deps.run("systemctl", ["--user", "daemon-reload"]); + } else { + deps.tryRun("launchctl", ["bootout", `gui/${deps.uid}/${LABEL}`]); + rmSync(path); + } + return path; +} diff --git a/src/cli/transcript.ts b/src/cli/transcript.ts new file mode 100644 index 00000000..e65af1ba --- /dev/null +++ b/src/cli/transcript.ts @@ -0,0 +1,134 @@ +// What a session's own transcript on this host adds to its record: the first quest skill it ran, which gives its +// role, and how long each blocking question waited on a person. No prompt or reply text leaves this module. +import { CODEX_ASK, CODEX_REPLY, claudeSkill, codexSkill, eachLine, parseLine } from "./messages"; + +/** What a transcript says about its session. */ +export interface TranscriptFacts { + /** The first `quest-*` skill it loaded, without the prefix's harness path, as `quest-start`. */ + skill: string | null; + /** Each blocking question, from when it was asked to when the person answered or dismissed it, epoch ms. */ + questions: { start: number; end: number }[]; +} + +const time = (record: Record, path: string): number => { + const at = typeof record.timestamp === "string" ? Date.parse(record.timestamp) : NaN; + if (Number.isNaN(at)) throw new Error(`${path} has a record without a timestamp`); + return at; +}; + +/** + * The facts of the session whose transcript files token-max listed. A question still open at the end of the + * transcript has no wait yet; the next record, sent when the session stops again, holds it. + */ +export function transcriptFacts(harness: string, files: string[]): TranscriptFacts { + const facts: TranscriptFacts = { skill: null, questions: [] }; + for (const path of files) { + if (harness === "claude") readClaude(path, facts); + else if (harness === "codex") readCodex(path, facts); + else throw new Error(`quest cannot read a ${harness} transcript`); + } + facts.questions.sort((a, b) => a.start - b.start); + return facts; +} + +const firstQuestSkill = (facts: TranscriptFacts, skill: string | null) => { + if (facts.skill === null && skill?.startsWith("quest-")) facts.skill = skill; +}; + +function readClaude(path: string, facts: TranscriptFacts): void { + const asked = new Map(); + eachLine(path, (line, complete) => { + const ask = line.includes('"AskUserQuestion"'); + if (!ask && !line.includes("Base directory for this skill") && !line.includes('"tool_use_id"')) return; + const record = parseLine(path, line, complete); + if (!record) return; + // A sub-agent's own file marks every record a sidechain, so only the file tells whose it is. + const isMeta = { ...record, isSidechain: undefined }; + firstQuestSkill(facts, claudeSkill(isMeta)); + const content = (record.message as { content?: unknown } | undefined)?.content; + if (!Array.isArray(content)) return; + for (const block of content as Record[]) { + if (record.type === "assistant" && block?.type === "tool_use" && block.name === "AskUserQuestion") + asked.set(String(block.id), time(record, path)); + else if (record.type === "user" && block?.type === "tool_result") { + const start = asked.get(String(block.tool_use_id)); + if (start === undefined) continue; + asked.delete(String(block.tool_use_id)); + facts.questions.push({ start, end: time(record, path) }); + } + } + }); +} + +function readCodex(path: string, facts: TranscriptFacts): void { + const asked = new Map(); + eachLine(path, (line, complete) => { + const call = /"call_id"\s*:\s*"([^"]+)"/.exec(line)?.[1]; + const relevant = + line.includes("request_user_input") || + line.includes("send_user_message_question_reply") || + line.includes('"skills"') || + (call !== undefined && asked.has(call)); + if (!relevant) return; + const record = parseLine(path, line, complete); + const item = record?.payload as Record | undefined; + if (record?.type !== "response_item" || !item) return; + if (item.type === "function_call" && CODEX_ASK.has(item.name as string)) { + asked.set(String(item.call_id), time(record, path)); + return; + } + if (item.type === "function_call_output" && item.call_id !== undefined) { + const start = asked.get(String(item.call_id)); + // The async variant's output only acknowledges the question; its answer arrives as a later turn. + if (start === undefined || item.output === undefined || isAcknowledgement(item.output)) return; + asked.delete(String(item.call_id)); + facts.questions.push({ start, end: time(record, path) }); + return; + } + if (item.role !== "user" || !Array.isArray(item.content)) return; + const sources = ( + (item.internal_chat_message_metadata_passthrough as Record | undefined) + ?.content_item_metadata as unknown[] | undefined + )?.map((each) => (each as { provenance?: { type?: unknown } } | null)?.provenance?.type); + (item.content as Record[]).forEach((part, i) => { + if (typeof part?.text !== "string") return; + if (sources?.[i] === "skills") firstQuestSkill(facts, codexSkill(part.text)); + const reply = CODEX_REPLY.exec(part.text); + if (!reply) return; + let each: unknown; + try { + each = JSON.parse(reply[1]!); + } catch { + throw new Error(`${path} has a question reply in a shape quest does not know`); + } + if (!Array.isArray(each)) throw new Error(`${path} has a question reply in a shape quest does not know`); + const calls = new Set(); + for (const one of each) { + try { + const ref = JSON.parse((one as { questionItemId?: string })?.questionItemId ?? ""); + if (Array.isArray(ref) && typeof ref[1] === "string") calls.add(ref[1]); + } catch { + throw new Error(`${path} has a question reply in a shape quest does not know`); + } + } + for (const id of calls) { + const start = asked.get(id); + if (start === undefined) continue; + asked.delete(id); + facts.questions.push({ start, end: time(record, path) }); + } + }); + }); +} + +/** Whether a Codex question call's output only says the question was sent, as `request_user_input_async` returns. */ +function isAcknowledgement(output: unknown): boolean { + if (typeof output !== "string") return false; + try { + const parsed = JSON.parse(output) as Record | null; + return parsed !== null && typeof parsed === "object" && !("answers" in parsed); + } catch { + // A cancelled or failed call returns an error message: the person was not waited on further. + return false; + } +} diff --git a/src/cli/usage.ts b/src/cli/usage.ts index c54a38b9..6531bee0 100644 --- a/src/cli/usage.ts +++ b/src/cli/usage.ts @@ -13,8 +13,19 @@ import { MERGER } from "./land-watch-fix"; import { githubRepository, requireRemote } from "./github"; import { type GitHub, type ObjectValue, type Repository, link } from "../github/client"; import { type OnlineReview, readOnlineReviews } from "../github/online-review"; -import { type Counts, type ReviewSummary, findingsText, parseReview, severityText } from "./review-record"; +import { + type Counts, + type ReviewSummary, + classifierSessions, + findingsText, + parseReview, + reviewSessions, + severityText, +} from "./review-record"; import { SEVERITIES } from "./review"; +import { ROLES, type Role, type SessionRecord, WAIT_KINDS } from "../core"; +import { buildRecords } from "./records"; +import { type LoggedWait, readWaits } from "./waits"; /** A session's tokens, and their cost at the model's API list price, which subscriptions do not bill. */ export interface Tokens { @@ -45,28 +56,21 @@ export interface Session { repositories: string[] | null; /** Each branch it named, with the first and last time, in epoch milliseconds. */ branches: { name: string; first: number; last: number }[]; - /** Epoch milliseconds of its first request in the window. */ + /** Epoch milliseconds of its first and last request in the window. */ start: number; + end: number; models: Spent[]; cost: number; /** False when its cost is a lower bound: a request went unpriced, or a line was unreadable or unfinished. */ complete: boolean; /** Transcript lines token-max could not read, which its cost leaves out. */ skippedLines: number; + /** Its transcript files, on its host. */ + files: string[]; } -export type Role = "implementing" | "iterating" | "reviewing" | "classifying"; - -/** One session attributed to a PR. */ -export interface SessionRow { - host: string | null; - harness: string; - session: string | null; - /** The session that spawned it, as a sub-agent's. */ - parent: string | null; - role: Role; - models: Spent[]; -} +/** One session's record, on a PR it worked on, and the host it was read on, or null for this one. */ +export type SessionRow = SessionRecord & { host: string | null }; /** One PR's record and the sessions that worked on it. */ export interface PullRow { number: number; @@ -226,6 +230,8 @@ export interface UsageOptions { export interface UsageDependencies extends ClaimDependencies { /** token-max's sessions since an instant on this host and each SSH destination, or null when it is not installed. */ sessions(since: Date, ssh: string[]): Session[] | null; + /** The waits logged on this host that ended since an instant, epoch ms. */ + waits(since: number): LoggedWait[]; } /** The install hint printed when token-max is missing. */ @@ -302,6 +308,7 @@ export function parseSessions(text: string): Session[] { last: time(b.last, "a branch's last time"), })), start: time(row.start, "a start"), + end: time(row.end, "an end"), models: models.map((m: Record) => ({ model: optional(m?.model, "a model"), effort: optional(m.effort, "an effort"), @@ -315,6 +322,9 @@ export function parseSessions(text: string): Session[] { cost: count(row.costUsd, "a cost"), complete, skippedLines: count(row.skippedLines, "a skipped line count"), + files: (Array.isArray(row.files) ? row.files : fail("a file list")).map((f: unknown) => + word(f, "a file"), + ), }; }); } @@ -340,7 +350,8 @@ export function fold(sessions: Session[]): { kept: Session[]; dropped: Session[] return { kept, dropped }; } -interface Pull { +/** A PR as `quest usage` reads it from GitHub. */ +export interface Pull { number: number; url: string; title: string; @@ -354,7 +365,8 @@ interface Pull { body: string; } -const key = (repository: Repository) => `${repository.owner}/${repository.repo}`.toLowerCase(); +/** `repository` as lowercase `owner/repo`. */ +export const key = (repository: Repository) => `${repository.owner}/${repository.repo}`.toLowerCase(); function pull(row: ObjectValue): Pull { const head = row.head as ObjectValue; @@ -375,7 +387,7 @@ function pull(row: ObjectValue): Pull { } /** The PRs of `repository` updated since `since`, newest update first, read until the listing passes it. */ -async function updatedPulls(client: GitHub, repository: Repository, since: number): Promise { +export async function updatedPulls(client: GitHub, repository: Repository, since: number): Promise { const pulls: Pull[] = []; let next: string | undefined = `${repositoryPath(repository)}/pulls?state=all&sort=updated&direction=desc&per_page=100`; @@ -493,29 +505,24 @@ function owners(sessions: Session[]): (session: Session) => string[] | null { } /** - * Attribute each session to the PR whose head branch it last named, among those still open when the session - * reached the branch and headed from a repository its host resolved for it, else to its parent's PR, as a sub-agent - * in a detached checkout, unless its repository is known to be another. Give it a role: classifying when a PR's review record names it as the risk classifier, - * reviewing when the record names it as a reviewer or Codex marks it a review, its parent's when both share the PR, - * as a review's sub-agents do, else implementing when its first request in the window came before its PR opened, - * and iterating otherwise. `evidence` holds every exported copy, folded ones included, since a dropped copy may be - * the only one naming its directory's repository. Returns the sessions it gave a PR in `rows`, and those on no PR here. + * The PR each session's own branch heads: the one whose head branch it last named, among those still open when the + * session reached the branch and headed from a repository its host resolved for it. `evidence` holds every exported + * copy, folded ones included, since a dropped copy may be the only one naming its directory's repository. */ function attribute( sessions: Session[], evidence: Session[], pulls: Pull[], - rows: Map, - recorded: { reviewers: Set; classifiers: Set }, -): { attributed: Session[]; unattributed: Session[] } { +): { own: Map; known: (session: Session) => string[] | null } { const byBranch = new Map(); for (const each of [...pulls].sort((a, b) => a.created - b.created)) byBranch.set(each.branch, [...(byBranch.get(each.branch) ?? []), each]); + const own = new Map(); const owned = owners(evidence); - const own = (session: Session): Pull | undefined => { + for (const session of sessions) { // Another repository may use the same branch name, so a session counts only where its repository is known. const known = owned(session); - if (known === null) return undefined; + if (known === null) continue; for (const seen of [...session.branches].sort((a, b) => b.last - a.last)) { // A branch can head a new PR after an earlier one closed: take the one open when the session reached it. const open = (byBranch.get(seen.name) ?? []).filter( @@ -526,69 +533,61 @@ function attribute( ); // A checkout with remotes for both a fork and its upstream cannot tell which one's branch it was on. if (new Set(open.map((each) => each.repository)).size > 1) continue; - if (open[0]) return open[0]; - } - return undefined; - }; - const id = (harness: string, session: string | null) => JSON.stringify([harness, session]); - const byId = new Map(sessions.filter((each) => each.id !== null).map((each) => [id(each.harness, each.id), each])); - type Placed = { pull: Pull; role: Role } | null; - const placed = new Map(); - const place = (session: Session, seen: Set): Placed => { - if (placed.has(session)) return placed.get(session)!; - const parent = session.parent === null ? undefined : byId.get(id(session.harness, session.parent)); - // A parent chain that loops back is cut there, so its sessions place by their own branches. - const above = parent === undefined || seen.has(parent) ? null : place(parent, new Set([...seen, session])); - // A session known to be in another repository keeps out of its parent's PR, as a sub-agent in another checkout. - const known = owned(session); - const inherited = - above && (known === null || (above.pull.repository !== null && known.includes(above.pull.repository))) - ? above.pull - : undefined; - const pull = own(session) ?? inherited; - let result: Placed = null; - if (pull) - result = { - pull, - role: - session.id !== null && recorded.classifiers.has(session.id) - ? "classifying" - : session.review || (session.id !== null && recorded.reviewers.has(session.id)) - ? "reviewing" - : above && above.pull === pull - ? above.role - : session.start < pull.created - ? "implementing" - : "iterating", - }; - placed.set(session, result); - return result; - }; - const attributed: Session[] = []; - const unattributed: Session[] = []; - for (const session of sessions) { - const found = place(session, new Set()); - if (!found) { - unattributed.push(session); - continue; + const found = open[0]; + if (found) { + own.set(session, found); + break; + } } - const row = rows.get(found.pull.number); - if (!row) continue; - attributed.push(session); - row.sessions.push({ - host: session.host, - harness: session.harness, - session: session.id, - parent: session.parent, - role: found.role, - models: session.models, - }); } - return { attributed, unattributed }; + return { own, known: owned }; } +/** What `sessionRecords` builds from. */ +export interface RecordSources { + /** The base repository as lowercase `owner/repo`. */ + repository: string; + /** The sessions to attribute, each once. */ + sessions: Session[]; + /** Every exported copy, folded ones included, for the repositories each directory belongs to. */ + evidence: Session[]; + /** The repository's PRs in the window. */ + pulls: Pull[]; + /** The waits logged on this host. */ + waits: LoggedWait[]; + /** The sessions to build records for; by default, those on a PR. */ + emit?: Set; + /** The window's start, epoch ms. */ + since: number; +} + +/** Each session's record, attributed to the PRs in the window, whose review records name their reviewers and classifier. */ +export function sessionRecords(sources: RecordSources): Map { + const { repository, sessions, evidence, pulls, waits, emit, since } = sources; + // Only the reviewer sessions are read, so no PR author can stop every session's record with a body this version + // cannot parse. + const reviewers = new Set(pulls.flatMap((pull) => reviewSessions(pull.body))); + const classifiers = new Set(pulls.flatMap((pull) => classifierSessions(pull.body))); + return buildRecords({ + repository, + sessions, + ...attribute(sessions, evidence, pulls), + reviewers, + classifiers, + waits, + emit, + since, + }); +} + +/** A row of a session with no ID, by its host and transcript files, which no two sessions share. */ +const anonymous = new WeakMap(); + +/** One session across the PR rows it appears on. */ +const identity = (session: SessionRow) => JSON.stringify([session.harness, session.session ?? anonymous.get(session)]); + function spendRows(pulls: PullRow[]): SpendRow[] { - const rows = new Map(); + const rows = new Map }>(); for (const each of pulls) for (const session of each.sessions) for (const spent of session.models) { @@ -603,19 +602,23 @@ function spendRows(pulls: PullRow[]): SpendRow[] { effort: spent.effort, sessions: 0, ...zero(), + seen: new Set(), }; rows.set(id, row); } - row.sessions++; - add(row, spent); + if (!row.seen.has(identity(session))) row.sessions++; + row.seen.add(identity(session)); + add(row, portion(session, spent)); } - const roles: Role[] = ["implementing", "iterating", "reviewing", "classifying"]; - return [...rows.values()].sort((a, b) => - order( - [a.size, roles.indexOf(a.role), a.harness, a.model, a.effort], - [b.size, roles.indexOf(b.role), b.harness, b.model, b.effort], - ), - ); + const roles: readonly Role[] = ROLES; + return [...rows.values()] + .map(({ seen: _, ...row }) => row) + .sort((a, b) => + order( + [a.size, roles.indexOf(a.role), a.harness, a.model, a.effort], + [b.size, roles.indexOf(b.role), b.harness, b.model, b.effort], + ), + ); } const noCounts = (): Omit => ({ @@ -693,6 +696,11 @@ export function share( return shares; } +/** A session's spend on one PR: on several, as a coordinator is, it counts toward each in equal shares. */ +const portion = (session: SessionRow, spent: Spent): Spent => + session.pulls.length > 1 ? { ...spent, ...part(spent, session.pulls.length) } : spent; +const shareOf = (session: SessionRow): Spent[] => session.models.map((spent) => portion(session, spent)); + /** One `n`th of `tokens`. */ function part(tokens: Tokens, n: number): Tokens { return { @@ -778,7 +786,7 @@ function accuracyRows(pulls: PullRow[]): AccuracyRow[] { ]; const writing = each.sessions .filter((session) => session.role === "implementing" || session.role === "iterating") - .flatMap((session) => session.models.map((spent) => ({ ...spent, role: session.role }))); + .flatMap((session) => shareOf(session).map((spent) => ({ ...spent, role: session.role }))); const shares = each.state === "merged" ? share(writing, distinct) : new Map(); for (const [i, implementer] of distinct.entries()) { @@ -845,7 +853,7 @@ function reviewerRows(pulls: PullRow[]): ReviewerRow[] { }; const reviewing = each.sessions .filter((session) => session.role === "reviewing") - .flatMap((session) => session.models.map((spent) => ({ ...spent, seat: named(session) }))); + .flatMap((session) => shareOf(session).map((spent) => ({ ...spent, seat: named(session) }))); for (const [i, spent] of share(reviewing, each.review)) { const to = row(each, each.review[i]!).spent; to.pulls++; @@ -874,6 +882,7 @@ function reviewerRows(pulls: PullRow[]): ReviewerRow[] { */ function armRows(pulls: PullRow[]): ArmRow[] { const rows = new Map(); + const seen = new Map>(); const latest = tallied(pulls, (each) => each.implementers.length > 0); for (const each of pulls) { const first = each.implementers[0]; @@ -887,9 +896,13 @@ function armRows(pulls: PullRow[]): ArmRow[] { rows.set(id, row); } outcome(row, each, latest); + const counted = seen.get(row) ?? new Set(); + seen.set(row, counted); for (const session of each.sessions) { - row.sessions++; - for (const spent of session.models) add(row.spend, spent); + // A session on several PRs of the arm, as a coordinator is, counts once, its shares adding up to its cost. + if (!counted.has(identity(session))) row.sessions++; + counted.add(identity(session)); + for (const spent of shareOf(session)) add(row.spend, spent); } } return [...rows.values()].sort((a, b) => @@ -986,12 +999,27 @@ export async function usage(root: string, options: UsageOptions, deps: UsageDepe let folded: Session[] = dropped; if (exported !== null) { // Every PR in the window competes for a session, so narrowing never hands one PR another's sessions. - const reviewers = new Set(allRows.flatMap((row) => row.review.flatMap((review) => review.session ?? []))); - const classifiers = new Set(allRows.flatMap((row) => row.risk?.session ?? [])); - const { attributed, unattributed: left } = attribute(kept, exported ?? [], all, byNumber, { - reviewers, - classifiers, + const records = sessionRecords({ + repository: key(repository), + sessions: kept, + evidence: exported, + pulls: all, + waits: deps.waits(since), + since, }); + const attributed: Session[] = []; + const left: Session[] = []; + for (const session of kept) { + const record = records.get(session); + const on = record?.pulls.flatMap((number) => byNumber.get(number) ?? []) ?? []; + for (const row of on) { + const each = { ...record!, host: session.host }; + if (each.session === null) anonymous.set(each, JSON.stringify([session.host, session.files])); + row.sessions.push(each); + } + if (on.length) attributed.push(session); + else if (!record?.pulls.length) left.push(session); + } const reported = narrowed ? attributed : [...attributed, ...left]; if (!narrowed) { unattributed = { sessions: left.length, ...zero() }; @@ -1050,6 +1078,31 @@ const percent = (part: number, whole: number) => (whole ? `${Math.round((100 * p const answers = (r: Omit) => `${r.findings} finding${r.findings === 1 ? "" : "s"}: ${r.accepted} accepted, ${r.declined} declined, ${r.unclassified} unclassified, ${r.unanswered} unanswered`; +/** Seconds under a minute, hours past two, else minutes. */ +const duration = (ms: number) => + ms < 60_000 + ? `${Math.round(ms / 1000)}s` + : ms >= 7_200_000 + ? `${(ms / 3_600_000).toFixed(1)}h` + : `${Math.round(ms / 60_000)}m`; + +/** + * The time a PR's sessions spent in each kind of wait, a shared session's in its share, and how many sessions' + * waits are unknown, as for those read on another host; empty when there is nothing to say. + */ +function waitTotals(sessions: SessionRow[]): string { + const totals = new Map(); + let unknown = 0; + for (const s of sessions) { + if (s.waits === null) unknown++; + for (const w of s.waits ?? []) + totals.set(w.kind, (totals.get(w.kind) ?? 0) + (Date.parse(w.end) - Date.parse(w.start)) / s.pulls.length); + } + const parts = WAIT_KINDS.filter((kind) => totals.has(kind)).map((kind) => `${kind} ${duration(totals.get(kind)!)}`); + if (unknown) parts.push(`unknown for ${unknown} session${unknown === 1 ? "" : "s"} on other hosts`); + return parts.join(", "); +} + /** Rows of cells, each column padded to its widest cell. */ function table(rows: string[][]): string[] { const widths = rows.reduce((w, row) => row.map((cell, i) => Math.max(w[i] ?? 0, cell.length)), []); @@ -1097,10 +1150,12 @@ export function usageLines(report: Usage): string[] { s.harness, m.model ?? "-", m.effort ?? "-", - tokens(m), + `${tokens(m)}${s.pulls.length > 1 ? `, 1/${s.pulls.length} here` : ""}`, ]), ); if (sessions.length) lines.push(" sessions:", ...table(sessions)); + const waited = waitTotals(row.sessions); + if (waited) lines.push(` waits: ${waited}`); } if (report.spend.length) lines.push( @@ -1217,5 +1272,5 @@ export function usageLines(report: Usage): string[] { /** The dependencies `quest usage` runs with. */ export function usageDependencies(github: ClaimDependencies): UsageDependencies { - return { ...github, sessions: runTokenMax }; + return { ...github, sessions: runTokenMax, waits: (since) => readWaits(since) }; } diff --git a/src/cli/waits.ts b/src/cli/waits.ts new file mode 100644 index 00000000..281a222e --- /dev/null +++ b/src/cli/waits.ts @@ -0,0 +1,136 @@ +// The waits `quest gate` and `quest wait` time for the agent session that ran them, kept on this host so a session's +// record can add them up. The harness names its session in each command's environment; nothing is logged outside one. +import { closeSync, constants, mkdirSync, openSync, readdirSync, readFileSync, rmSync, writeSync } from "node:fs"; +import { homedir } from "node:os"; +import { join } from "node:path"; + +import { type SessionWait, WAIT_KINDS } from "../core"; + +/** A session a harness names in a command's environment. */ +export interface EnvSession { + harness: string; + session: string; +} + +/** One logged wait, with every session its environment named and the directory it ran in. */ +export interface LoggedWait extends SessionWait { + sessions: EnvSession[]; + directory: string; +} + +/** Days of waits kept, beyond the longest session a record is expected to span. */ +const KEEP_DAYS = 90; +const DAY = 86_400_000; + +/** + * Every session the environment names. A Claude Code sub-agent's commands carry its parent's ID, and an agent one + * harness starts inherits the other's variable, so a wait may name several; the record builder picks among them. + */ +export function envSessions(env: NodeJS.ProcessEnv = process.env): EnvSession[] { + const found: EnvSession[] = []; + if (env.CLAUDE_CODE_SESSION_ID) found.push({ harness: "claude", session: env.CLAUDE_CODE_SESSION_ID }); + if (env.CODEX_THREAD_ID) found.push({ harness: "codex", session: env.CODEX_THREAD_ID }); + return found; +} + +/** Where the waits are logged, one file per UTC day. */ +export function waitsDirectory(env: NodeJS.ProcessEnv = process.env): string { + return join(env.XDG_STATE_HOME || join(homedir(), ".local", "state"), "quest", "waits"); +} + +const day = (ms: number) => new Date(ms).toISOString().slice(0, 10); + +/** + * Append `wait` to today's log in one write, so concurrent writers leave only whole lines, when the environment + * names a session; drop days past `KEEP_DAYS`. Throws when it cannot write. + */ +export function logWait( + wait: SessionWait, + options: { env?: NodeJS.ProcessEnv; cwd?: string; now?: number } = {}, +): void { + const env = options.env ?? process.env; + const sessions = envSessions(env); + if (sessions.length === 0) return; + const now = options.now ?? Date.now(); + const dir = waitsDirectory(env); + try { + mkdirSync(dir, { recursive: true, mode: 0o700 }); + const line = Buffer.from( + `${JSON.stringify({ ...wait, sessions, directory: options.cwd ?? process.cwd() } satisfies LoggedWait)}\n`, + ); + const fd = openSync( + join(dir, `${day(now)}.jsonl`), + constants.O_WRONLY | constants.O_APPEND | constants.O_CREAT | constants.O_NOFOLLOW, + 0o600, + ); + try { + const written = writeSync(fd, line); + if (written !== line.length) throw new Error(`wrote ${written} of ${line.length} bytes`); + } finally { + closeSync(fd); + } + const oldest = day(now - KEEP_DAYS * DAY); + for (const name of readdirSync(dir)) + if (/^\d{4}-\d\d-\d\d\.jsonl$/.test(name) && name.slice(0, 10) < oldest) + rmSync(join(dir, name), { force: true }); + } catch (error) { + throw new Error(`cannot log the wait in ${dir}: ${error instanceof Error ? error.message : String(error)}`); + } +} + +/** The waits logged on this host that ended at or after `since`, failing on a line it cannot read. */ +export function readWaits(since: number, env: NodeJS.ProcessEnv = process.env): LoggedWait[] { + const dir = waitsDirectory(env); + let names: string[]; + try { + names = readdirSync(dir); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return []; + throw error; + } + const first = day(since); + const waits: LoggedWait[] = []; + for (const name of names.sort()) { + if (!/^\d{4}-\d\d-\d\d\.jsonl$/.test(name) || name.slice(0, 10) < first) continue; + const path = join(dir, name); + const lines = readFileSync(path, "utf8").split("\n"); + lines.forEach((line, i) => { + if (line === "") return; + const wait = parseLogged(line); + if (wait === null) { + // A writer appends whole lines, so only a last line still being written may be cut short. + if (i === lines.length - 1) return; + throw new Error(`${path}:${i + 1} is not a wait quest logged`); + } + if (Date.parse(wait.end) >= since) waits.push(wait); + }); + } + return waits; +} + +function parseLogged(line: string): LoggedWait | null { + let value: Record; + try { + value = JSON.parse(line) as Record; + } catch { + return null; + } + const iso = (v: unknown) => typeof v === "string" && !Number.isNaN(Date.parse(v)); + const sessions = value?.sessions; + if ( + !WAIT_KINDS.includes(value?.kind as SessionWait["kind"]) || + !iso(value.start) || + !iso(value.end) || + !(value.until === null || typeof value.until === "string") || + typeof value.directory !== "string" || + !Array.isArray(sessions) || + !sessions.every((s) => typeof s?.harness === "string" && typeof s?.session === "string") + ) + return null; + return value as unknown as LoggedWait; +} + +/** The kind of a `quest wait` that ended on `event`: the merge queue's when the PR merged or left the queue. */ +export function pullWaitKind(event: string | null): SessionWait["kind"] { + return event === "merged" || event === "removed_from_merge_queue" ? "merge-queue" : "pull"; +} diff --git a/src/core/index.ts b/src/core/index.ts index ea091740..cd18aee5 100644 --- a/src/core/index.ts +++ b/src/core/index.ts @@ -94,3 +94,15 @@ export { latestHold, heldToken, } from "./hold"; +export { + type ModelSpend, + type RecordUpload, + ROLES, + type Role, + type SessionRecord, + type SessionWait, + WAIT_KINDS, + type WaitKind, + parseRecord, + parseUpload, +} from "./record"; diff --git a/src/core/record.ts b/src/core/record.ts new file mode 100644 index 00000000..73523210 --- /dev/null +++ b/src/core/record.ts @@ -0,0 +1,191 @@ +// The record of one agent session that `quest` sends to Kixel Code, which stores exactly this shape, and that +// `quest usage` prints. One session is one record: a resend under the same harness and session ID replaces it. A +// host's one sender uploads a session's copies in order, so the latest upload is the newest copy. + +/** What a session did for its PRs. */ +export const ROLES = [ + "planning", + "implementing", + "iterating", + "reviewing", + "classifying", + "merging", + "coordinating", +] as const; +export type Role = (typeof ROLES)[number]; + +/** + * What a session waited on, timed by the tool that waited: queued in `quest gate`, running a gated command, waiting + * in `quest wait` for PR activity, which `until` names, or for the merge queue to merge or drop the PR, and waiting + * on a person's answer to a blocking question. + */ +export const WAIT_KINDS = ["gate-queue", "gate-run", "pull", "merge-queue", "person"] as const; +export type WaitKind = (typeof WAIT_KINDS)[number]; + +/** One timed wait, its instants ISO 8601 UTC. */ +export interface SessionWait { + kind: WaitKind; + start: string; + end: string; + /** For a `pull` or `merge-queue` wait, the first activity `quest wait` printed, as `checks:passed`; null on timeout. */ + until: string | null; +} + +/** One model and effort's tokens in a session, and their cost at API list price in USD. */ +export interface ModelSpend { + model: string | null; + effort: string | null; + input: number; + cacheWrite: number; + cacheRead: number; + output: number; + cost: number; + /** False when the cost is a lower bound: a request went unpriced or a transcript line was unreadable. */ + complete: boolean; +} + +/** One agent session, as `quest` sends it. */ +export interface SessionRecord { + /** `claude` or `codex`. */ + harness: string; + /** + * The harness's own session ID; a Claude sub-agent's is its agent ID. Null when its transcript names none, which + * `quest usage` still reports but no upload carries, since the store replaces a record by its ID. + */ + session: string | null; + /** The session that spawned it, or null. */ + parent: string | null; + /** The base repository as lowercase `owner/repo`. */ + repository: string; + /** Its PR's head branch, else the branch it last named, so a session before its PR opens joins it later. */ + branch: string | null; + /** The quest file its branch claims, as `quest/m0/foo.md`. */ + quest: string | null; + /** + * The PRs it worked on: its own, else its parent's, else, for a coordinator, every PR its sub-agents worked on. + * Its cost counts equally toward each, so a PR's cost is the sum of each record's cost over its PR count. + */ + pulls: number[]; + role: Role; + models: ModelSpend[]; + cost: number; + complete: boolean; + /** Its first and last request, ISO 8601 UTC. */ + start: string; + end: string; + /** Its timed waits, oldest first; null when they could not be read, as for a session on another host. */ + waits: SessionWait[] | null; +} + +/** The body of a record upload. */ +export interface RecordUpload { + records: SessionRecord[]; +} + +const fail = (what: string): never => { + throw new Error(`malformed session record: ${what}`); +}; +const isObject = (value: unknown): value is Record => + typeof value === "object" && value !== null && !Array.isArray(value); +const text = (value: unknown, what: string): string => (typeof value === "string" && value !== "" ? value : fail(what)); +const maybe = (value: unknown, what: string): string | null => (value === null ? null : text(value, what)); +const count = (value: unknown, what: string): number => + typeof value === "number" && Number.isFinite(value) && value >= 0 ? value : fail(what); +const flag = (value: unknown, what: string): boolean => (typeof value === "boolean" ? value : fail(what)); +const instant = (value: unknown, what: string): string => { + const at = text(value, what); + return /^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d(\.\d+)?Z$/.test(at) && !Number.isNaN(Date.parse(at)) ? at : fail(what); +}; +const oneOf = (value: unknown, options: readonly T[], what: string): T => + options.includes(value as T) ? (value as T) : fail(what); +const exact = (value: Record, keys: string[], what: string) => { + const extra = Object.keys(value).find((key) => !keys.includes(key)); + if (extra !== undefined) fail(`${what} has an unknown field ${extra}`); +}; + +/** `value` as a record, failing on any missing, extra, or ill-typed field, so a store never keeps one in part. */ +export function parseRecord(value: unknown): SessionRecord { + if (!isObject(value)) return fail("not an object"); + const keys = [ + "harness", + "session", + "parent", + "repository", + "branch", + "quest", + "pulls", + "role", + "models", + "cost", + "complete", + "start", + "end", + "waits", + ]; + exact(value, keys, "the record"); + const pulls = Array.isArray(value.pulls) ? value.pulls : fail("pulls"); + const models = Array.isArray(value.models) ? value.models : fail("models"); + const waits = value.waits === null ? null : Array.isArray(value.waits) ? value.waits : fail("waits"); + const repository = text(value.repository, "repository"); + if (!/^[a-z\d._-]+\/[a-z\d._-]+$/.test(repository)) fail("repository is not a lowercase owner/repo"); + const quest = maybe(value.quest, "quest"); + if (quest !== null && !/^quest\/.+\.md$/.test(quest)) fail("quest is not a quest path"); + const record: SessionRecord = { + harness: text(value.harness, "harness"), + session: maybe(value.session, "session"), + parent: maybe(value.parent, "parent"), + repository, + branch: maybe(value.branch, "branch"), + quest, + pulls: pulls.map((n) => (Number.isSafeInteger(n) && (n as number) > 0 ? (n as number) : fail("a PR number"))), + role: oneOf(value.role, ROLES, "role"), + models: models.map((m: unknown) => { + if (!isObject(m)) return fail("a model"); + exact(m, ["model", "effort", "input", "cacheWrite", "cacheRead", "output", "cost", "complete"], "a model"); + return { + model: maybe(m.model, "a model name"), + effort: maybe(m.effort, "an effort"), + input: count(m.input, "input tokens"), + cacheWrite: count(m.cacheWrite, "cache write tokens"), + cacheRead: count(m.cacheRead, "cache read tokens"), + output: count(m.output, "output tokens"), + cost: count(m.cost, "a model's cost"), + complete: flag(m.complete, "a model's completeness"), + }; + }), + cost: count(value.cost, "cost"), + complete: flag(value.complete, "completeness"), + start: instant(value.start, "start"), + end: instant(value.end, "end"), + waits: + waits?.map((w: unknown) => { + if (!isObject(w)) return fail("a wait"); + exact(w, ["kind", "start", "end", "until"], "a wait"); + const wait: SessionWait = { + kind: oneOf(w.kind, WAIT_KINDS, "a wait's kind"), + start: instant(w.start, "a wait's start"), + end: instant(w.end, "a wait's end"), + until: maybe(w.until, "a wait's until"), + }; + if (Date.parse(wait.end) < Date.parse(wait.start)) fail("a wait ends before it starts"); + if (wait.until !== null && wait.kind !== "pull" && wait.kind !== "merge-queue") + fail(`a ${wait.kind} wait names what ended it`); + return wait; + }) ?? null, + }; + if (new Set(record.pulls).size !== record.pulls.length) fail("a PR listed twice"); + if (Date.parse(record.end) < Date.parse(record.start)) fail("it ends before it starts"); + return record; +} + +/** An upload's body, failing whole on any malformed record, one naming no session, or a session sent twice. */ +export function parseUpload(value: unknown): RecordUpload { + if (!isObject(value)) return fail("the upload is not an object"); + exact(value, ["records"], "the upload"); + if (!Array.isArray(value.records)) return fail("the upload has no record list"); + const records = value.records.map(parseRecord); + if (records.some((r) => r.session === null)) fail("an uploaded record names no session"); + const ids = new Set(records.map((r) => `${r.harness}\0${r.session}`)); + if (ids.size !== records.length) fail("the upload holds a session twice"); + return { records }; +} diff --git a/test/hooks.test.ts b/test/hooks.test.ts index 136a56cd..6be27f2d 100644 --- a/test/hooks.test.ts +++ b/test/hooks.test.ts @@ -247,6 +247,8 @@ test.each([ const stop = { hook_event_name: event, cwd, + session_id: "s1", + transcript_path: transcript, [key]: transcript, stop_hook_active: true, background_tasks: [ @@ -257,16 +259,18 @@ test.each([ { id: "a1", type: "subagent", status: "running", description: "review" }, ], }; - const output = JSON.parse(hook(stop)); + // A user's own state, so a sender's failure on this host is never said here; this checkout has no origin. + const deps = { env: { XDG_STATE_HOME: tempDir(), XDG_CONFIG_HOME: tempDir() } }; + const output = JSON.parse(hook(stop, deps)); expect(output.decision).toBe("block"); expect(output.reason).toContain("- cd repo && quest gate just test"); expect(output.reason).not.toContain("just check"); // Without a transcript, each running gate counts as the agent's. - expect(JSON.parse(hook({ ...stop, [key]: "" })).reason).toContain("just check"); + expect(JSON.parse(hook({ ...stop, [key]: "" }, deps)).reason).toContain("just check"); // It lets the turn end once the gate ends, and outside a repository that uses Quest. - expect(hook({ ...stop, background_tasks: stop.background_tasks.slice(1) })).toBe(""); - expect(hook({ ...stop, cwd: checkout(null) })).toBe(""); - expect(() => hook({ hook_event_name: event, cwd })).toThrow("background_tasks"); + expect(hook({ ...stop, background_tasks: stop.background_tasks.slice(1) }, deps)).toBe(""); + expect(hook({ ...stop, cwd: checkout(null) }, deps)).toBe(""); + expect(() => hook({ hook_event_name: event, cwd }, deps)).toThrow("background_tasks"); }); test("hook refuses malformed input and events it does not handle", () => { diff --git a/test/records.test.ts b/test/records.test.ts new file mode 100644 index 00000000..a419bcc2 --- /dev/null +++ b/test/records.test.ts @@ -0,0 +1,496 @@ +import { appendFileSync, existsSync, mkdirSync, readdirSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +import { describe, expect, test } from "vitest"; + +import { type SessionRecord, parseRecord, parseUpload } from "../src/core"; +import { buildRecords } from "../src/cli/records"; +import { transcriptFacts } from "../src/cli/transcript"; +import type { Pull, Session } from "../src/cli/usage"; +import { type LoggedWait, logWait, pullWaitKind, readWaits, waitsDirectory } from "../src/cli/waits"; +import { tempDir } from "./fixture"; + +const record = (extra: Partial = {}): SessionRecord => ({ + harness: "claude", + session: "s1", + parent: null, + repository: "acme/proj", + branch: "quest/m0/a", + quest: "quest/m0/a.md", + pulls: [10], + role: "implementing", + models: [ + { + model: "claude-opus-5-5", + effort: "high", + input: 1, + cacheWrite: 2, + cacheRead: 3, + output: 4, + cost: 0.5, + complete: true, + }, + ], + cost: 0.5, + complete: true, + start: "2026-11-06T10:00:00.000Z", + end: "2026-11-06T11:00:00.000Z", + waits: [ + { kind: "pull", start: "2026-11-06T10:10:00.000Z", end: "2026-11-06T10:20:00.000Z", until: "checks:passed" }, + ], + ...extra, +}); + +describe("parseRecord", () => { + test("orders instants by time, not by their text", () => { + expect(parseRecord(record({ start: "2026-11-06T10:00:00Z", end: "2026-11-06T10:00:00.001Z" })).end).toBe( + "2026-11-06T10:00:00.001Z", + ); + }); + + test("keeps a well-formed record exactly", () => { + const sent = record(); + expect(parseRecord(JSON.parse(JSON.stringify(sent)))).toEqual(sent); + expect(parseRecord(record({ waits: null, pulls: [], branch: null, quest: null }))).toMatchObject({ + waits: null, + }); + }); + + test.each([ + ["an unknown field", { ...record(), extra: 1 }, "unknown field extra"], + ["an unknown role", record({ role: "lurking" as never }), "role"], + ["a negative cost", record({ cost: -1 }), "cost"], + ["a PR listed twice", record({ pulls: [10, 10] }), "twice"], + ["a local-time instant", record({ start: "2026-11-06T10:00:00" }), "start"], + [ + "a wait ending before it starts", + record({ + waits: [ + { kind: "person", start: "2026-11-06T10:20:00.000Z", end: "2026-11-06T10:10:00.000Z", until: null }, + ], + }), + "ends before", + ], + [ + "a gate wait naming an event", + record({ + waits: [ + { + kind: "gate-run", + start: "2026-11-06T10:10:00.000Z", + end: "2026-11-06T10:20:00.000Z", + until: "merged", + }, + ], + }), + "names what ended it", + ], + [ + "an end before its start in finer precision", + record({ start: "2026-11-06T10:00:00.001Z", end: "2026-11-06T10:00:00Z" }), + "ends before it starts", + ], + ["a mixed-case repository", record({ repository: "Acme/Proj" }), "lowercase"], + ["a quest that is not a path", record({ quest: "a.md" }), "quest path"], + ])("refuses %s", (_, value, message) => { + expect(() => parseRecord(value)).toThrow(message); + }); + + test("an upload fails whole on one bad record or a session sent twice", () => { + expect(parseUpload({ records: [record(), record({ session: "s2" })] }).records).toHaveLength(2); + expect(() => parseUpload({ records: [record(), record({ session: "s2", cost: Number.NaN })] })).toThrow("cost"); + expect(() => parseUpload({ records: [record(), record()] })).toThrow("twice"); + expect(() => parseUpload({ records: [], more: true })).toThrow("unknown field"); + // A record may name no session, but no upload carries one, since the store replaces records by session. + expect(parseRecord(record({ session: null })).session).toBeNull(); + expect(() => parseUpload({ records: [record({ session: null })] })).toThrow("names no session"); + }); +}); + +describe("wait log", () => { + const env = (extra: NodeJS.ProcessEnv = {}) => ({ XDG_STATE_HOME: tempDir(), ...extra }); + const wait = { + kind: "gate-run" as const, + start: "2026-11-06T10:00:00.000Z", + end: "2026-11-06T10:05:00.000Z", + until: null, + }; + + test("logs only in an agent session, naming every session the environment does", () => { + const outside = env(); + logWait(wait, { env: outside, cwd: "/w", now: Date.parse(wait.end) }); + expect(existsSync(waitsDirectory(outside))).toBe(false); + const inside = env({ CLAUDE_CODE_SESSION_ID: "c1", CODEX_THREAD_ID: "x1" }); + logWait(wait, { env: inside, cwd: "/w", now: Date.parse(wait.end) }); + expect(readWaits(0, inside)).toEqual([ + { + ...wait, + sessions: [ + { harness: "claude", session: "c1" }, + { harness: "codex", session: "x1" }, + ], + directory: "/w", + }, + ]); + }); + + test("reads waits that ended since an instant, tolerating only a cut-off last line", () => { + const e = env({ CODEX_THREAD_ID: "x1" }); + logWait(wait, { env: e, cwd: "/w", now: Date.parse(wait.end) }); + const later = { ...wait, start: "2026-11-07T10:00:00.000Z", end: "2026-11-07T10:05:00.000Z" }; + logWait(later, { env: e, cwd: "/w", now: Date.parse(later.end) }); + expect(readWaits(Date.parse("2026-11-07T00:00:00Z"), e).map((w) => w.end)).toEqual([later.end]); + const today = join(waitsDirectory(e), "2026-11-07.jsonl"); + appendFileSync(today, '{"kind":"gate-r'); + expect(readWaits(0, e)).toHaveLength(2); + appendFileSync(today, '\n{"kind":"gate-run"}\n'); + expect(() => readWaits(0, e)).toThrow("2026-11-07.jsonl:2"); + }); + + test("drops days past what it keeps", () => { + const e = env({ CODEX_THREAD_ID: "x1" }); + const dir = waitsDirectory(e); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, "2026-01-01.jsonl"), ""); + writeFileSync(join(dir, "notes.txt"), ""); + logWait(wait, { env: e, cwd: "/w", now: Date.parse("2026-11-06T10:05:00Z") }); + expect(readdirSync(dir).sort()).toEqual(["2026-11-06.jsonl", "notes.txt"]); + }); + + test("a quest wait that ends on a merge or a dequeue waited on the merge queue", () => { + expect(pullWaitKind("merged")).toBe("merge-queue"); + expect(pullWaitKind("removed_from_merge_queue")).toBe("merge-queue"); + expect(pullWaitKind("checks:passed")).toBe("pull"); + expect(pullWaitKind(null)).toBe("pull"); + }); +}); + +const line = (value: object) => `${JSON.stringify(value)}\n`; + +describe("transcriptFacts", () => { + test("reads a Claude transcript's first quest skill and how long each question waited", () => { + const path = join(tempDir(), "s.jsonl"); + const skill = (name: string, at: string) => + line({ + type: "user", + isMeta: true, + isSidechain: true, + timestamp: at, + message: { + content: [{ type: "text", text: `Base directory for this skill: /x/.claude/skills/${name}` }], + }, + }); + writeFileSync( + path, + line({ type: "summary" }) + + skill("simplify", "2026-11-06T10:00:00Z") + + skill("quest-start", "2026-11-06T10:01:00Z") + + skill("quest-review", "2026-11-06T10:02:00Z") + + line({ + type: "assistant", + timestamp: "2026-11-06T10:03:00Z", + message: { content: [{ type: "tool_use", id: "q1", name: "AskUserQuestion", input: {} }] }, + }) + + line({ + type: "user", + timestamp: "2026-11-06T10:13:00Z", + message: { content: [{ type: "tool_result", tool_use_id: "q1", content: "x" }] }, + toolUseResult: { questions: [], answers: {} }, + }) + + // A question still open has no wait yet. + line({ + type: "assistant", + timestamp: "2026-11-06T10:20:00Z", + message: { content: [{ type: "tool_use", id: "q2", name: "AskUserQuestion", input: {} }] }, + }), + ); + expect(transcriptFacts("claude", [path])).toEqual({ + skill: "quest-start", + questions: [{ start: Date.parse("2026-11-06T10:03:00Z"), end: Date.parse("2026-11-06T10:13:00Z") }], + }); + }); + + test("reads Codex's blocking and asynchronous questions", () => { + const path = join(tempDir(), "rollout.jsonl"); + const item = (at: string, payload: object) => line({ timestamp: at, type: "response_item", payload }); + writeFileSync( + path, + line({ timestamp: "2026-11-06T10:00:00Z", type: "session_meta", payload: { id: "t1" } }) + + item("2026-11-06T10:00:01Z", { + type: "message", + role: "user", + content: [{ type: "input_text", text: "quest-iterate" }], + internal_chat_message_metadata_passthrough: { + content_item_metadata: [{ provenance: { type: "skills" } }], + }, + }) + + item("2026-11-06T10:01:00Z", { + type: "function_call", + name: "request_user_input", + call_id: "c1", + arguments: "{}", + }) + + item("2026-11-06T10:02:00Z", { + type: "function_call_output", + call_id: "c1", + output: '{"answers":{}}', + }) + + item("2026-11-06T10:03:00Z", { + type: "function_call", + name: "request_user_input_async", + call_id: "c2", + arguments: "{}", + }) + + item("2026-11-06T10:03:01Z", { + type: "function_call_output", + call_id: "c2", + output: '{"status":"sent"}', + }) + + item("2026-11-06T10:09:00Z", { + type: "message", + role: "user", + content: [ + { + type: "input_text", + text: `${JSON.stringify([{ questionItemId: JSON.stringify(["x", "c2", 0]), answer: "A" }])}`, + }, + ], + }), + ); + expect(transcriptFacts("codex", [path])).toEqual({ + skill: "quest-iterate", + questions: [ + { start: Date.parse("2026-11-06T10:01:00Z"), end: Date.parse("2026-11-06T10:02:00Z") }, + { start: Date.parse("2026-11-06T10:03:00Z"), end: Date.parse("2026-11-06T10:09:00Z") }, + ], + }); + }); +}); + +describe("buildRecords", () => { + const at = (time: string) => Date.parse(`2026-11-06T${time}Z`); + const session = (id: string, extra: Partial = {}): Session => ({ + host: null, + harness: "claude", + id, + parent: null, + review: false, + cwd: "/w", + repositories: ["acme/proj"], + branches: [], + start: at("10:00:00"), + end: at("12:00:00"), + models: [ + { model: "m", effort: "high", input: 1, cacheWrite: 0, cacheRead: 0, output: 1, cost: 1, complete: true }, + ], + cost: 1, + complete: true, + skippedLines: 0, + files: [], + ...extra, + }); + const pull = (number: number, branch: string, created: string): Pull => ({ + number, + url: `https://github.com/acme/proj/pull/${number}`, + title: "t", + branch, + repository: "acme/proj", + state: "open", + draft: false, + created: at(created), + closed: null, + body: "", + }); + const logged = (kind: LoggedWait["kind"], start: string, sessions: string[], directory = "/w"): LoggedWait => ({ + kind, + start: `2026-11-06T${start}.000Z`, + end: `2026-11-06T${start.replace(/:\d\d$/, ":59")}.000Z`, + until: null, + sessions: sessions.map((s) => ({ harness: "claude", session: s })), + directory, + }); + + test("a coordinator shares its sub-agents' PRs, and a helper sub-agent its parent's", () => { + const coordinator = session("run"); + const a = session("a", { parent: "run", cwd: "/w/.worktrees/a", start: at("10:10:00"), end: at("10:50:00") }); + const b = session("b", { parent: "run", cwd: "/w/.worktrees/b", start: at("10:20:00"), end: at("11:00:00") }); + const helper = session("h", { parent: "a", start: at("10:15:00"), end: at("10:16:00") }); + const loose = session("loose", { + branches: [{ name: "quest-plan/x", first: at("10:00:00"), last: at("10:01:00") }], + }); + const p1 = pull(1, "quest/m0/a", "10:30:00"); + const p2 = pull(2, "quest/m0/b", "10:05:00"); + const records = buildRecords({ + repository: "acme/proj", + sessions: [coordinator, a, b, helper, loose], + own: new Map([ + [a, p1], + [b, p2], + ]), + reviewers: new Set(), + classifiers: new Set(), + waits: [ + logged("gate-run", "10:40:00", ["run"], "/w/.worktrees/a/src"), + logged("gate-queue", "10:25:00", ["run"], "/w"), + logged("pull", "10:30:00", ["run"], "/w/.worktrees"), + // Outside any session's window that is running there, it stays with the coordinator. + logged("gate-run", "11:30:00", ["run"], "/w/.worktrees/b"), + ], + emit: new Set([coordinator, a, b, helper, loose]), + since: 0, + }); + const of = (s: Session) => records.get(s)!; + expect(of(coordinator)).toMatchObject({ pulls: [1, 2], role: "coordinating", branch: null, quest: null }); + expect(of(a)).toMatchObject({ pulls: [1], role: "implementing", branch: "quest/m0/a", quest: "quest/m0/a.md" }); + expect(of(b)).toMatchObject({ pulls: [2], role: "iterating" }); + expect(of(helper)).toMatchObject({ pulls: [1], role: "implementing", parent: "a" }); + expect(of(loose)).toMatchObject({ pulls: [], role: "planning", branch: "quest-plan/x", quest: null }); + expect(of(a).waits!.map((w) => w.kind)).toEqual(["gate-run"]); + expect(of(b).waits).toEqual([]); + // Two sub-agents are running where the wait ran, so the coordinator keeps it. + expect(of(coordinator).waits!.map((w) => [w.kind, w.start.slice(11, 16)])).toEqual([ + ["gate-queue", "10:25"], + ["pull", "10:30"], + ["gate-run", "11:30"], + ]); + }); + + test("reviewers, skills, and the innermost of several sessions a wait names", () => { + const transcript = join(tempDir(), "t.jsonl"); + writeFileSync( + transcript, + line({ + type: "user", + isMeta: true, + timestamp: "2026-11-06T10:00:00Z", + message: { content: [{ type: "text", text: "Base directory for this skill: /s/quest-merge" }] }, + }), + ); + const merger = session("m", { files: [transcript] }); + const reviewer = session("r", { harness: "codex", start: at("10:30:00") }); + const remote = session("far", { host: "box" }); + const p = pull(3, "quest/m0/c", "09:00:00"); + const records = buildRecords({ + repository: "acme/proj", + sessions: [merger, reviewer, remote], + own: new Map([ + [merger, p], + [reviewer, p], + [remote, p], + ]), + reviewers: new Set(["r"]), + classifiers: new Set(["far"]), + waits: [ + { + ...logged("gate-run", "10:40:00", []), + sessions: [ + { harness: "claude", session: "m" }, + { harness: "codex", session: "r" }, + ], + }, + ], + since: 0, + }); + expect(records.get(merger)).toMatchObject({ role: "merging", waits: [] }); + expect(records.get(remote)!.role).toBe("classifying"); + expect(records.get(reviewer)!.role).toBe("reviewing"); + expect(records.get(reviewer)!.waits!.map((w) => w.kind)).toEqual(["gate-run"]); + expect(records.get(remote)!.waits).toBeNull(); + for (const r of records.values()) expect(parseRecord(JSON.parse(JSON.stringify(r)))).toEqual(r); + }); + + test("a sub-agent known to be in another repository keeps out of its parent's PR", () => { + const parent = session("p"); + const here = session("h", { parent: "p" }); + const elsewhere = session("e", { parent: "p" }); + const records = buildRecords({ + repository: "acme/proj", + sessions: [parent, here, elsewhere], + own: new Map([[parent, pull(1, "quest/m0/a", "09:00:00")]]), + known: (s) => (s === elsewhere ? ["acme/other"] : s === here ? null : ["acme/proj"]), + reviewers: new Set(), + classifiers: new Set(), + waits: [], + since: 0, + }); + expect(records.get(here)!.pulls).toEqual([1]); + expect(records.has(elsewhere)).toBe(false); + }); + + test("without emit, only sessions on a PR get a record", () => { + const on = session("on"); + const off = session("off"); + const records = buildRecords({ + repository: "acme/proj", + sessions: [on, off], + own: new Map([[on, pull(1, "quest/m0/a", "09:00:00")]]), + reviewers: new Set(), + classifiers: new Set(), + waits: [], + since: 0, + }); + expect([...records.keys()]).toEqual([on]); + }); + + test("a session with no ID keeps its PR, and a wait begun before the window still counts", () => { + const anonymous = { ...session("x"), id: null }; + // Its first request in the window comes after the wait ended. + const named = session("n", { start: at("10:45:00") }); + const records = buildRecords({ + repository: "acme/proj", + sessions: [anonymous, named], + own: new Map([ + [anonymous, pull(1, "quest/m0/a", "09:00:00")], + [named, pull(1, "quest/m0/a", "09:00:00")], + ]), + reviewers: new Set(), + classifiers: new Set(), + // Begun at 10:10, before the window opened at 10:20 and before the session's first request in it. + waits: [{ ...logged("pull", "10:10:00", ["n"]), end: "2026-11-06T10:40:00.000Z" }], + since: at("10:20:00"), + }); + expect(records.get(anonymous)).toMatchObject({ session: null, pulls: [1] }); + expect(records.get(named)!.waits!.map((w) => w.kind)).toEqual(["pull"]); + }); + + test("a person's wait is windowed, and an unreadable transcript leaves waits unknown", () => { + const transcript = join(tempDir(), "t.jsonl"); + const ask = (id: string, asked: string, answered: string) => + line({ + type: "assistant", + timestamp: `2026-11-06T${asked}Z`, + message: { content: [{ type: "tool_use", id, name: "AskUserQuestion", input: {} }] }, + }) + + line({ + type: "user", + timestamp: `2026-11-06T${answered}Z`, + message: { content: [{ type: "tool_result", tool_use_id: id }] }, + }); + writeFileSync(transcript, ask("q1", "08:00:00", "08:30:00") + ask("q2", "10:10:00", "10:20:00")); + const asked = session("asked", { files: [transcript] }); + const broken = join(tempDir(), "b.jsonl"); + writeFileSync(broken, '{"type":"assistant","AskUserQuestion" not json\n{"type":"summary"}\n'); + const skipped = session("skipped", { files: [broken], skippedLines: 1, complete: false }); + const p = pull(1, "quest/m0/a", "09:00:00"); + const input = { + repository: "acme/proj", + sessions: [asked, skipped], + own: new Map([ + [asked, p], + [skipped, p], + ]), + reviewers: new Set(), + classifiers: new Set(), + waits: [], + since: at("09:00:00"), + }; + const records = buildRecords(input); + expect(records.get(asked)!.waits).toEqual([ + { kind: "person", start: "2026-11-06T10:10:00.000Z", end: "2026-11-06T10:20:00.000Z", until: null }, + ]); + expect(records.get(skipped)!.waits).toBeNull(); + // A line token-max read but quest cannot is a shape quest does not know. + const unknown = { ...skipped, skippedLines: 0 }; + expect(() => buildRecords({ ...input, sessions: [unknown], own: new Map([[unknown, p]]) })).toThrow("not JSON"); + }); +}); diff --git a/test/send.test.ts b/test/send.test.ts new file mode 100644 index 00000000..7e7b3029 --- /dev/null +++ b/test/send.test.ts @@ -0,0 +1,442 @@ +import { execFileSync } from "node:child_process"; +import { mkdirSync, readdirSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +import { expect, test } from "vitest"; + +import { REFERENCE_LINE, parseUpload } from "../src/core"; +import { hook } from "../src/cli/hooks"; +import { type SendDependencies, queue, runSender, sendPass, tokensPath } from "../src/cli/send"; +import type { Session } from "../src/cli/usage"; +import { tempDir } from "./fixture"; + +const ENDPOINT = "https://kixel.example/records"; + +/** A checkout that uses Quest, with a GitHub origin, naming `endpoint` when given. */ +function checkout(endpoint?: unknown): string { + const dir = tempDir(); + execFileSync("git", ["init", "-q", dir]); + execFileSync("git", ["-C", dir, "remote", "add", "origin", "https://github.com/acme/proj.git"]); + writeFileSync(join(dir, "AGENTS.md"), `${REFERENCE_LINE}\n`); + if (endpoint !== undefined) writeFileSync(join(dir, ".quest.json"), JSON.stringify({ records: endpoint })); + return dir; +} + +/** A user's state and config, with a token for `ENDPOINT` unless told otherwise, and a sender running. */ +function home(token: string | null = "secret"): NodeJS.ProcessEnv { + const env = { XDG_STATE_HOME: tempDir(), XDG_CONFIG_HOME: tempDir() }; + mkdirSync(join(env.XDG_STATE_HOME, "quest", "records"), { recursive: true }); + writeFileSync(senderPid(env), String(process.pid)); + if (token !== null) { + mkdirSync(join(env.XDG_CONFIG_HOME, "quest")); + writeFileSync(tokensPath(env), JSON.stringify({ [ENDPOINT]: token })); + } + return env; +} + +function transcript(): string { + const path = join(tempDir(), "s.jsonl"); + writeFileSync(path, `{"type":"summary"}\n{"type":"user","timestamp":"2026-11-06T10:00:00Z"}\n`); + return path; +} + +const senderPid = (env: NodeJS.ProcessEnv) => join(env.XDG_STATE_HOME!, "quest", "records", "sender.pid"); + +const stopped = (cwd: string) => ({ harness: "claude", session: "s1", transcript: transcript(), cwd }); + +test("a checkout naming no endpoint, or one without the user's token, queues nothing and says so once", () => { + const env = home(null); + const bare = checkout(); + expect(queue(stopped(bare), env)).toEqual({ + notices: [expect.stringContaining("acme/proj names no Kixel Code endpoint in .quest.json")], + queued: false, + }); + expect(queue(stopped(bare), env)).toEqual({ notices: [], queued: false }); + const named = checkout(ENDPOINT); + expect(queue(stopped(named), env).notices).toEqual([expect.stringContaining(`holds no token for ${ENDPOINT}`)]); + expect(queue(stopped(named), env).notices).toEqual([]); + // Outside a checkout that uses Quest, nothing is said. + expect(queue(stopped(tempDir()), env)).toEqual({ notices: [], queued: false }); +}); + +test("a checkout's endpoint must be HTTPS, or HTTP to this host", () => { + const env = home(); + expect(() => queue(stopped(checkout("http://kixel.example/records")), env)).toThrow("not an HTTPS URL"); + expect(() => queue(stopped(checkout(7)), env)).toThrow("not a URL"); + expect(queue(stopped(checkout("http://127.0.0.1:8080/records")), env).notices).toEqual([ + expect.stringContaining("holds no token for http://127.0.0.1:8080/records"), + ]); +}); + +/** A token-max session in acme/proj on `branch`. */ +const session = (id: string, branch: string, extra: Partial = {}): Session => ({ + host: null, + harness: "claude", + id, + parent: null, + review: false, + cwd: "/w", + repositories: ["acme/proj"], + branches: [{ name: branch, first: Date.parse("2026-11-06T10:00:00Z"), last: Date.parse("2026-11-06T10:30:00Z") }], + start: Date.parse("2026-11-06T10:00:00Z"), + end: Date.parse("2026-11-06T11:00:00Z"), + models: [{ model: "m", effort: "high", input: 1, cacheWrite: 0, cacheRead: 0, output: 1, cost: 2, complete: true }], + cost: 2, + complete: true, + skippedLines: 0, + files: [], + ...extra, +}); + +/** One user's clock, shared by every sender built for it, so a test moves time across passes. */ +const clocks = new WeakMap(); +function clockOf(env: NodeJS.ProcessEnv): { now: number } { + if (!clocks.has(env)) clocks.set(env, { now: Date.parse("2026-11-06T12:00:00Z") }); + return clocks.get(env)!; +} + +/** Sender dependencies whose endpoint answers `status`, recording each upload and token-max's window. */ +function sender( + env: NodeJS.ProcessEnv, + status: number | (() => number), + sessions: Session[] | null, + during: () => void = () => {}, +) { + const uploads: { token: string; body: unknown }[] = []; + const windows: Date[] = []; + const slept: number[] = []; + const clock = clockOf(env); + const json = (value: unknown) => + new Response(JSON.stringify(value), { headers: { "content-type": "application/json" } }); + const github = async (input: string | URL | Request) => { + const url = new URL(String(input)); + if (url.pathname === "/repos/acme/proj") + return json({ + owner: { login: "acme" }, + name: "proj", + default_branch: "main", + clone_url: "https://github.com/acme/proj.git", + ssh_url: "git@github.com:acme/proj.git", + fork: false, + }); + if (url.pathname === "/repos/acme/proj/pulls") + return json([ + { + number: 7, + html_url: "https://github.com/acme/proj/pull/7", + title: "A", + head: { ref: "quest/m0/a", repo: { full_name: "acme/proj" } }, + state: "open", + draft: true, + merged_at: null, + created_at: "2026-11-06T10:10:00Z", + updated_at: "2026-11-06T11:00:00Z", + closed_at: null, + body: "", + }, + ]); + return new Response("not found", { status: 404 }); + }; + const deps: SendDependencies = { + env, + github: () => ({ + fetch: github as typeof fetch, + now: Date.now, + sleep: async () => {}, + token: "t", + warn: () => {}, + }), + sessions: (since) => { + windows.push(since); + return sessions; + }, + waits: () => [], + now: () => clock.now, + sleep: async (ms) => { + slept.push(ms); + clock.now += ms; + }, + interval: 60_000, + fetch: (async (input: string | URL | Request, init?: RequestInit) => { + expect(String(input)).toBe(ENDPOINT); + uploads.push({ + token: String((init?.headers as Record).authorization), + body: JSON.parse(String(init?.body)), + }); + during(); + return new Response("", { status: typeof status === "number" ? status : status() }); + }) as typeof fetch, + }; + return { deps, uploads, windows, slept, clock }; +} + +const pending = (env: NodeJS.ProcessEnv) => readdirSync(join(env.XDG_STATE_HOME!, "quest", "records", "pending")); + +test("the sender posts each queued session with its sub-agents, from its transcript's start", async () => { + const env = home(); + const root = checkout(ENDPOINT); + expect(queue(stopped(root), env)).toEqual({ notices: [], queued: true }); + const sessions = [ + session("s1", "quest/m0/a"), + session("sub", "quest/m0/a", { parent: "s1" }), + session("other", "quest/m0/a"), + ]; + const { deps, uploads, windows } = sender(env, 200, sessions); + expect(await sendPass(deps)).toEqual([{ endpoint: ENDPOINT, repository: "acme/proj", records: 2, refused: false }]); + expect(windows).toEqual([new Date("2026-11-06T10:00:00Z")]); + expect(uploads).toHaveLength(1); + expect(uploads[0]!.token).toBe("Bearer secret"); + const { records } = parseUpload(uploads[0]!.body); + expect(records.map((r) => [r.session, r.pulls, r.role, r.quest])).toEqual([ + ["s1", [7], "implementing", "quest/m0/a.md"], + ["sub", [7], "implementing", "quest/m0/a.md"], + ]); + expect(pending(env)).toEqual([]); +}); + +test("a failed upload stays queued and is said once; a refused token stops sending", async () => { + const env = home(); + const root = checkout(ENDPOINT); + queue(stopped(root), env); + const failing = sender(env, 500, [session("s1", "quest/m0/a")]); + await expect(sendPass(failing.deps)).rejects.toThrow("answered 500"); + failing.clock.now += 60_000; + expect(pending(env)).toHaveLength(1); + const again = queue(stopped(root), env); + expect(again.notices).toEqual([expect.stringContaining("answered 500")]); + expect(queue(stopped(root), env).notices).toEqual([]); + + const refusing = sender(env, 403, [session("s1", "quest/m0/a")]); + expect(await sendPass(refusing.deps)).toEqual([ + { endpoint: ENDPOINT, repository: "acme/proj", records: 1, refused: true }, + ]); + expect(pending(env)).toEqual([]); + expect(queue(stopped(root), env)).toEqual({ + notices: [expect.stringContaining("refused this host's token (403)")], + queued: false, + }); + // A new token is tried again. + writeFileSync(tokensPath(env), JSON.stringify({ [ENDPOINT]: "fresh" })); + expect(queue(stopped(root), env).queued).toBe(true); +}); + +test("the sender fails loud without token-max, keeping the queue", async () => { + const env = home(); + queue(stopped(checkout(ENDPOINT)), env); + await expect(sendPass(sender(env, 200, null).deps)).rejects.toThrow("token-max is not installed"); + expect(pending(env)).toHaveLength(1); +}); + +test("a failing group backs off, doubling, without widening the others' window", async () => { + const env = home(); + const root = checkout(ENDPOINT); + queue(stopped(root), env); + const failing = sender(env, 500, [session("s1", "quest/m0/a")]); + await expect(sendPass(failing.deps)).rejects.toThrow("tried again in 1 minutes"); + // Within its wait, the group sits the pass out: no token-max run, no upload. + failing.clock.now += 30_000; + expect(await sendPass(failing.deps)).toEqual([]); + expect(failing.windows).toHaveLength(1); + failing.clock.now += 30_000; + await expect(sendPass(failing.deps)).rejects.toThrow("tried again in 2 minutes"); + failing.clock.now += 60_000; + expect(await sendPass(failing.deps)).toEqual([]); + // Another repository's stop, queued later, is sent from its own start meanwhile. + const other = checkout(ENDPOINT); + execFileSync("git", ["-C", other, "remote", "set-url", "origin", "https://github.com/acme/other.git"]); + writeFileSync(join(other, "later.jsonl"), `{"type":"user","timestamp":"2026-11-06T11:00:00Z"}\n`); + queue({ ...stopped(other), session: "s2", transcript: join(other, "later.jsonl") }, env); + const others = sender(env, 200, [session("s2", "main", { repositories: ["acme/other"] })]); + // The stand-in GitHub knows no acme/other, so that group fails too, after token-max read only its window. + await expect(sendPass(others.deps)).rejects.toThrow("acme/other"); + expect(others.windows).toEqual([new Date("2026-11-06T11:00:00Z")]); + // Once its wait ends, the first group is tried again and sent, and only the other's stop stays queued. + failing.clock.now += 60_000; + const fixed = sender(env, 200, [session("s1", "quest/m0/a")]); + await expect(sendPass(fixed.deps)).rejects.toThrow("acme/other"); + expect(fixed.uploads).toHaveLength(1); + expect(pending(env)).toHaveLength(1); +}); + +test("an unreadable backoff file reads as none, so it blocks no group", async () => { + const env = home(); + queue(stopped(checkout(ENDPOINT)), env); + const failing = sender(env, 500, [session("s1", "quest/m0/a")]); + await expect(sendPass(failing.deps)).rejects.toThrow("tried again in 1 minutes"); + const dir = join(env.XDG_STATE_HOME!, "quest", "records", "backoff"); + const [name] = readdirSync(dir); + expect(readdirSync(dir)).toEqual([name]); + // As a crash mid-write would leave it, were it written in place. + writeFileSync(join(dir, name!), '{"failures":'); + const sent = sender(env, 200, [session("s1", "quest/m0/a")]); + expect(await sendPass(sent.deps)).toHaveLength(1); + expect(readdirSync(dir)).toEqual([]); +}); + +test("an upload this version's own parseUpload refuses fails on the host, never posted", async () => { + const env = home(); + queue(stopped(checkout(ENDPOINT)), env); + const { deps, uploads } = sender(env, 200, [session("s1", "quest/m0/a", { cost: -1 })]); + await expect(sendPass(deps)).rejects.toThrow("its own parseUpload refuses"); + expect(uploads).toEqual([]); + expect(pending(env)).toHaveLength(1); +}); + +test("a session token-max no longer lists is dropped and said, without holding back the rest", async () => { + const env = home(); + const root = checkout(ENDPOINT); + queue(stopped(root), env); + queue({ ...stopped(root), session: "gone" }, env); + const { deps, uploads } = sender(env, 200, [session("s1", "quest/m0/a")]); + expect(await sendPass(deps)).toEqual([{ endpoint: ENDPOINT, repository: "acme/proj", records: 1, refused: false }]); + expect(parseUpload(uploads[0]!.body).records.map((r) => r.session)).toEqual(["s1"]); + expect(pending(env)).toEqual([]); + expect(queue(stopped(root), env).notices).toEqual([ + expect.stringContaining("lists no claude session gone on this host, so its record was dropped"), + ]); +}); + +test("a sub-agent is sent only where its own checkout routes it", async () => { + const env = home(); + const root = checkout(ENDPOINT); + queue(stopped(root), env); + const other = checkout(ENDPOINT); + execFileSync("git", ["-C", other, "remote", "set-url", "origin", "https://github.com/acme/fork.git"]); + const sessions = [ + session("s1", "quest/m0/a"), + // Live checkouts: this one, another of the same origin and endpoint, one opted out, and another origin's. + session("here", "quest/m0/a", { parent: "s1", cwd: root }), + session("twin", "quest/m0/a", { parent: "s1", cwd: checkout(ENDPOINT) }), + session("optedOut", "quest/m0/a", { parent: "s1", cwd: checkout() }), + // A deleted folder routes by the checkout still holding its nearest existing parent. + session("hereGone", "quest/m0/a", { parent: "s1", cwd: join(root, "deleted", "sub") }), + session("optedOutGone", "quest/m0/a", { parent: "s1", cwd: join(checkout(), "deleted", "sub") }), + // A deleted nested clone of another repository, inside this checkout. + session("nestedGone", "main", { + parent: "s1", + cwd: join(root, "vendor", "other"), + repositories: ["acme/other"], + }), + session("fork", "quest/m0/a", { parent: "s1", cwd: other, repositories: ["acme/fork", "acme/proj"] }), + // Deleted checkouts: every repository resolved must be this one's. + session("gone", "quest/m0/a", { parent: "s1" }), + session("goneFork", "quest/m0/a", { parent: "s1", repositories: ["acme/fork", "acme/proj"] }), + session("unknown", "quest/m0/a", { parent: "s1", repositories: null }), + ]; + const { deps, uploads } = sender(env, 200, sessions); + await sendPass(deps); + expect(parseUpload(uploads[0]!.body).records.map((r) => r.session)).toEqual([ + "s1", + "here", + "twin", + "hereGone", + "gone", + ]); +}); + +test("an origin that a URL rewrite points at GitHub is read as configured", () => { + const env = home(); + const root = checkout(ENDPOINT); + execFileSync("git", ["-C", root, "remote", "set-url", "origin", "gh:acme/proj"]); + execFileSync("git", ["-C", root, "config", "url.https://github.com/.insteadOf", "gh:"]); + expect(queue(stopped(root), env).queued).toBe(true); + execFileSync("git", ["-C", root, "config", "--unset", "url.https://github.com/.insteadOf"]); + execFileSync("git", ["-C", root, "remote", "set-url", "origin", "https://github.com/acme/proj.git"]); + execFileSync("git", ["-C", root, "config", "url.git@github-work:.insteadOf", "https://github.com/"]); + expect(queue(stopped(root), env).queued).toBe(true); +}); + +test("Claude Code's and Codex's Stop queue the session", () => { + const env = home(); + const cwd = checkout(ENDPOINT); + const deps = { env }; + const base = { hook_event_name: "Stop", cwd, session_id: "c1", transcript_path: transcript() }; + expect(hook({ ...base, background_tasks: [] }, deps)).toBe(""); + expect(hook({ ...base, session_id: "x1", turn_id: "t1" }, deps)).toBe(""); + expect(pending(env)).toHaveLength(2); + // A sub-agent goes with its parent, so its stop queues nothing. + expect(hook({ ...base, hook_event_name: "SubagentStop", background_tasks: [] }, deps)).toBe(""); + expect(pending(env)).toHaveLength(2); + // What it cannot queue, it says instead of failing the hook. + writeFileSync(join(cwd, ".quest.json"), "{"); + expect(JSON.parse(hook({ ...base, background_tasks: [] }, deps)).systemMessage).toContain( + "cannot queue this session's record", + ); + expect(() => hook({ hook_event_name: "Stop", cwd, turn_id: "t1" }, deps)).toThrow("session_id"); + // An ephemeral Codex session keeps no transcript, so its stop queues nothing. + expect(hook({ ...base, session_id: "e1", transcript_path: null, turn_id: "t1" }, deps)).toBe(""); +}); + +test("the sender passes every interval, records its PID, and outlives a failed pass", async () => { + const env = home(); + writeFileSync(senderPid(env), ""); + const root = checkout(ENDPOINT); + queue(stopped(root), env); + let status = 500; + const { deps, uploads, slept, clock } = sender(env, () => status, [session("s1", "quest/m0/a")]); + const log: string[] = []; + deps.sleep = async (ms) => { + slept.push(ms); + clock.now += ms; + status = 200; + if (slept.length === 3) throw new Error("stopped"); + }; + await expect(runSender(deps, (line) => log.push(line))).rejects.toThrow("stopped"); + expect(readFileSync(senderPid(env), "utf8")).toBe(String(process.pid)); + expect(slept).toEqual([60_000, 60_000, 60_000]); + // The failed pass keeps the session; the next sends it, and the last finds the queue empty. + expect(uploads).toHaveLength(2); + expect(log).toEqual([expect.stringContaining("answered 500"), `sent 1 records of acme/proj to ${ENDPOINT}`]); + expect(pending(env)).toEqual([]); +}); + +test("a stop with no sender running queues the session and says so once per outage", () => { + const env = home(); + const root = checkout(ENDPOINT); + writeFileSync(senderPid(env), "999999999"); + const down = queue(stopped(root), env); + expect(down).toEqual({ + notices: [expect.stringContaining("no session record sender runs on this host")], + queued: true, + }); + expect(down.notices[0]).toContain("quest records install"); + expect(queue(stopped(root), env)).toEqual({ notices: [], queued: true }); + expect(pending(env)).toHaveLength(2); + // A sender that ran again and stopped is a new outage. + writeFileSync(senderPid(env), "999999998"); + expect(queue(stopped(root), env).notices).toHaveLength(1); + writeFileSync(senderPid(env), String(process.pid)); + expect(queue(stopped(root), env).notices).toEqual([]); +}); + +test("a stop queued while its earlier record uploads stays queued", async () => { + const env = home(); + const root = checkout(ENDPOINT); + queue(stopped(root), env); + const first = sender(env, 200, [session("s1", "quest/m0/a")], () => { + if (first.uploads.length === 1) queue(stopped(root), env); + }); + await sendPass(first.deps); + expect(pending(env)).toHaveLength(1); + const next = sender(env, 200, [session("s1", "quest/m0/a")]); + await sendPass(next.deps); + expect(next.uploads).toHaveLength(1); + expect(pending(env)).toEqual([]); +}); + +test("a queued session for an endpoint that refused its token is dropped unsent", async () => { + const env = home(); + const root = checkout(ENDPOINT); + queue(stopped(root), env); + const refusing = sender(env, 403, [session("s1", "quest/m0/a")]); + // Queued before the refusal, so queue() could not drop it. + const dir = join(env.XDG_STATE_HOME!, "quest", "records", "pending"); + const [name] = readdirSync(dir); + const copy = readFileSync(join(dir, name!), "utf8"); + await sendPass(refusing.deps); + writeFileSync(join(dir, name!), copy); + const later = sender(env, 200, [session("s1", "quest/m0/a")]); + await sendPass(later.deps); + expect(later.uploads).toEqual([]); + expect(pending(env)).toEqual([]); +}); diff --git a/test/service.test.ts b/test/service.test.ts new file mode 100644 index 00000000..64609d2e --- /dev/null +++ b/test/service.test.ts @@ -0,0 +1,73 @@ +import { chmodSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +import { expect, test } from "vitest"; + +import { installSender, serviceDefinition, uninstallSender } from "../src/cli/service"; +import { tempDir } from "./fixture"; + +/** A user whose PATH holds an executable `quest` in a directory named `bin`, with a `%` and `$` to escape. */ +function user(): NodeJS.ProcessEnv { + const bin = join(tempDir(), "b%in$x"); + mkdirSync(bin); + writeFileSync(join(bin, "quest"), "#!/bin/sh\n"); + chmodSync(join(bin, "quest"), 0o755); + return { HOME: tempDir(), XDG_CONFIG_HOME: tempDir(), XDG_STATE_HOME: tempDir(), PATH: `relative:${bin}:/usr/bin` }; +} + +function deps(platform: NodeJS.Platform, env: NodeJS.ProcessEnv) { + const ran: string[] = []; + return { + ran, + deps: { + platform, + env, + uid: 501, + run: (command: string, args: string[]) => void ran.push([command, ...args].join(" ")), + tryRun: (command: string, args: string[]) => void ran.push(`try ${[command, ...args].join(" ")}`), + }, + }; +} + +test("on Linux, the sender is a systemd user service running quest from PATH, escaped", () => { + const env = user(); + const { deps: linux, ran } = deps("linux", env); + const path = installSender(linux); + expect(path).toBe(join(env.XDG_CONFIG_HOME!, "systemd", "user", "quest-records.service")); + const unit = readFileSync(path, "utf8"); + const bin = env.PATH!.split(":")[1]!; + expect(unit).toContain(`ExecStart="${bin.replace("%", "%%").replace("$", () => "$$")}/quest" records sender`); + expect(unit).toContain(`Environment="PATH=${env.PATH!.replace("%", "%%")}"`); + expect(unit).toContain(`Environment="XDG_STATE_HOME=${env.XDG_STATE_HOME}"`); + expect(unit).toContain(`Environment="XDG_CONFIG_HOME=${env.XDG_CONFIG_HOME}"`); + expect(unit).not.toContain("CODEX_HOME"); + expect(unit).toContain("Restart=always"); + expect(ran).toEqual([ + "systemctl --user daemon-reload", + "systemctl --user enable quest-records.service", + "systemctl --user restart quest-records.service", + ]); + ran.length = 0; + expect(uninstallSender(linux)).toBe(path); + expect(existsSync(path)).toBe(false); + expect(ran).toEqual(["systemctl --user disable --now quest-records.service", "systemctl --user daemon-reload"]); + expect(uninstallSender(linux)).toBeNull(); +}); + +test("on macOS, the sender is a launchd agent, replaced on reinstall", () => { + const env: NodeJS.ProcessEnv = { ...user(), PATH: `${user().PATH}:/a&b` }; + const { deps: mac, ran } = deps("darwin", env); + const path = installSender(mac); + expect(path).toBe(join(env.HOME!, "Library", "LaunchAgents", "dev.moq.quest.records.plist")); + const plist = readFileSync(path, "utf8"); + expect(plist).toContain("recordssender"); + expect(plist).toContain("/a&b"); + expect(plist).toContain(`XDG_STATE_HOME${env.XDG_STATE_HOME}`); + expect(plist).toContain("KeepAlive"); + expect(ran).toEqual(["try launchctl bootout gui/501/dev.moq.quest.records", `launchctl bootstrap gui/501 ${path}`]); +}); + +test("installing fails loud without quest on PATH or a supported service manager", () => { + expect(() => serviceDefinition("linux", { ...user(), PATH: "/nonexistent" })).toThrow("quest is not on PATH"); + expect(() => installSender(deps("win32", user()).deps)).toThrow("which win32 lacks"); +}); diff --git a/test/usage.test.ts b/test/usage.test.ts index fa1dd57f..0c57ce84 100644 --- a/test/usage.test.ts +++ b/test/usage.test.ts @@ -7,8 +7,9 @@ import { describe, expect, test } from "vitest"; import { implementer } from "../src/cli/models-record"; import { fixes } from "../src/cli/land-fixer"; import type { Risk, Round, Tally } from "../src/cli/review"; -import { parseReview, reviewRecord } from "../src/cli/review-record"; +import { classifierSessions, parseReview, reviewRecord, reviewSessions } from "../src/cli/review-record"; import { type Session, causes, fold, parseSessions, share, usage, usageLines } from "../src/cli/usage"; +import type { LoggedWait } from "../src/cli/waits"; import { tempDir } from "./fixture"; const THREAD = "01a1b8e8-8fca-7793-99d1-062f41f8a7b0"; @@ -60,7 +61,7 @@ const exported = (session: string, model: string, effort: string, cost: number, costUsd: cost, complete: true, skippedLines: 0, - files: ["/home/me/.claude/projects/-work/x.jsonl"], + files: [], ...extra, }); @@ -90,6 +91,7 @@ describe("parseSessions", () => { }, ], start: Date.parse("2026-11-06T10:05:00Z"), + end: Date.parse("2026-11-06T10:06:00Z"), models: [ { model: "gpt-6.1-sol", @@ -105,6 +107,7 @@ describe("parseSessions", () => { cost: 0.5, complete: true, skippedLines: 0, + files: [], }); // A session token-max could not resolve a repository, model, or ID for keeps the gap as null. const [unknown] = parse({ @@ -332,6 +335,23 @@ describe("PR records", () => { const settledAll = [tally("codex", 3, { high: 1, medium: 1, low: 2 }, 3, 1), tallies[1]!]; expect(reviewRecord({ rounds: clean, tallies: settledAll, risk })).toContain(`- Head: ${"b".repeat(40)}\n`); expect(fixes(body)).toEqual(["codex, gpt-6.1-sol, medium effort"]); + // The reviewer sessions read even from a record whose other lines this version cannot parse. + const newer = body.replace(/^- Rounds: 3;/m, "- Rounds: 3; first round 1 (medium 1);"); + expect(() => parseReview(newer)).toThrow("Rounds line is malformed"); + expect(reviewSessions(newer)).toEqual([THREAD, "thread-2", CLAUDE]); + expect(classifierSessions(body)).toEqual([]); + const classified = reviewRecord({ + rounds: [round], + tallies, + risk: { + ...risk, + classified: { + ...risk.classified, + by: { kind: "model", harness: "codex", model: "gpt-6.1-sol", effort: "low", session: "k1" }, + }, + }, + }); + expect(classifierSessions(`## Review\n\n${classified}\n`)).toEqual(["k1"]); }); test("a record written before tallies has none, and a Rounds line that does not add up fails", () => { @@ -736,7 +756,7 @@ describe("usage", () => { return new Response("not found", { status: 404 }); }; const calls: [Date, string[]][] = []; - const deps = (exported: Session[] | null) => ({ + const deps = (exported: Session[] | null, waits: LoggedWait[] = []) => ({ github: () => ({ fetch: fetch as typeof globalThis.fetch, now: Date.now, @@ -749,10 +769,24 @@ describe("usage", () => { calls.push([since, ssh]); return exported; }, + waits: () => waits, }); return { root, deps, sessions, calls, requests }; } + test("counts each session with no ID apart in the spend rollup", async () => { + const { root, deps, sessions } = setup(); + const known = sessions.find((s) => s.id === CLAUDE)!; + // Same host and span, told apart by their transcripts. + const anonymous = [1, 2].map((n) => ({ ...known, id: null, files: [`/t/${n}.jsonl`] })); + const report = await usage(root, { since: "2026-11-05" }, deps([...sessions, ...anonymous])); + const spent = known.models[0]!; + const row = report.spend.find( + (s) => s.role === "implementing" && s.model === spent.model && s.effort === (spent.effort ?? "unknown"), + )!; + expect([row.sessions, row.cost]).toEqual([3, 3 * known.cost]); + }); + test("ties each session to its PR and role, and rolls up spend and accuracy", async () => { const { root, deps, sessions, calls } = setup(); const report = await usage(root, { since: "2026-11-05", ssh: ["box"] }, deps(sessions)); @@ -1002,6 +1036,20 @@ describe("usage", () => { branches: [{ name: "quest/m0/b1", first: "2026-11-06T10:05:00Z", last: "2026-11-06T10:06:00Z" }], cwd: "/b1", }), + // A coordinator whose sub-agents worked on both arm-B PRs counts once there, at its whole cost. + exported("44444444-4444-4444-4444-444444444444", "claude-opus-5-5", "high", 4, { + branches: [], + cwd: "/c", + }), + ...["b1", "b2"].map((branch, i) => + exported(`5555555${i}-5555-5555-5555-555555555555`, "claude-opus-5-5", "high", 1, { + parent: "44444444-4444-4444-4444-444444444444", + branches: [ + { name: `quest/m0/${branch}`, first: "2026-11-06T10:05:00Z", last: "2026-11-06T10:06:00Z" }, + ], + cwd: `/${branch}`, + }), + ), ], }); const report = await usage(root, { since: "2026-11-05" }, deps(sessions)); @@ -1019,7 +1067,7 @@ describe("usage", () => { ]), ).toEqual([ ["M", "A", "claude-opus-5-5", "medium", 1, 1, 2, 0, 0], - ["M", "B", "claude-opus-5-5", "high", 2, 2, 4, 1, 3], + ["M", "B", "claude-opus-5-5", "high", 2, 2, 4, 4, 9], ]); const twenty = report.pulls.find((p) => p.number === 20)!; expect(twenty.implementers.map((i) => i.arm)).toEqual(["B", "B"]); @@ -1027,7 +1075,7 @@ describe("usage", () => { expect(text).toContain(" implementer: Claude Code, claude-opus-5-5, high effort (XS, arm B)\n"); expect(text).toContain("Writer A/B by size and arm:\n"); expect(text).toContain( - " M B Claude Code claude-opus-5-5 high 2 PRs, 2 merged, 0 handed back, 2 reviewed: rounds 2.0, findings 2 (medium 2): fixed 2, refuted 0, open 0, first rounds 0: 0; 0 later fixes; 1 sessions, in 10, cache 20 written 30 read, out 40, ~$3.00\n", + " M B Claude Code claude-opus-5-5 high 2 PRs, 2 merged, 0 handed back, 2 reviewed: rounds 2.0, findings 2 (medium 2): fixed 2, refuted 0, open 0, first rounds 0: 0; 0 later fixes; 4 sessions, in 40, cache 80 written 120 read, out 160, ~$9.00\n", ); }); diff --git a/vitest.config.ts b/vitest.config.ts index 3d66f802..b3489c41 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -19,7 +19,8 @@ export default defineConfig({ // Git reads none of the caller's own configuration, such as commit signing // with a key or program the fixtures cannot reach; each fixture sets its identity. // Git for Windows maps "/dev/null" to its null device but rejects os.devNull's "\\.\nul". - env: { GIT_CONFIG_GLOBAL: "/dev/null" }, + // An agent session's ID would log the gates tests run as that session's waits (src/cli/waits.ts). + env: { GIT_CONFIG_GLOBAL: "/dev/null", CLAUDE_CODE_SESSION_ID: "", CODEX_THREAD_ID: "" }, ...(live ? { include: ["test/live/**/*.test.ts"], testTimeout: 30 * 60_000, hookTimeout: 60_000 } : {