From 966d9df0514b272409dc81b9d54edba24987960c Mon Sep 17 00:00:00 2001 From: Daniel Ochoa Date: Mon, 19 Oct 2026 16:38:32 -0500 Subject: [PATCH 1/5] feat(vibe): Add codex-fleet-monitor skill (#308) - Background watcher for a fleet of Codex App Server threads (local and one ssh host) plus their pull requests; exits with a diff on the first change - Reads liveness from each thread's last turn and item count, so a turn left open by a silent usage limit shows as STALLED - vibe 1.0.57, README and changelog entries Testing: pytest plugins/ (2430 passed), ruff, plugin-manifests typecheck and tests, node --check and zsh -n on the scripts Risks: None identified, the skill only reads thread and PR state --- CHANGELOG.md | 5 ++ plugins/vibe/.claude-plugin/plugin.json | 2 +- plugins/vibe/.codex-plugin/plugin.json | 2 +- plugins/vibe/README.md | 10 +++ .../vibe/skills/codex-fleet-monitor/SKILL.md | 68 +++++++++++++++++++ .../codex-fleet-monitor/agents/openai.yaml | 4 ++ .../codex-fleet-monitor/scripts/classify.sh | 16 +++++ .../codex-fleet-monitor/scripts/liveness.mjs | 37 ++++++++++ .../codex-fleet-monitor/scripts/snapshot.sh | 37 ++++++++++ .../codex-fleet-monitor/scripts/watch.sh | 12 ++++ 10 files changed, 191 insertions(+), 2 deletions(-) create mode 100644 plugins/vibe/skills/codex-fleet-monitor/SKILL.md create mode 100644 plugins/vibe/skills/codex-fleet-monitor/agents/openai.yaml create mode 100755 plugins/vibe/skills/codex-fleet-monitor/scripts/classify.sh create mode 100644 plugins/vibe/skills/codex-fleet-monitor/scripts/liveness.mjs create mode 100755 plugins/vibe/skills/codex-fleet-monitor/scripts/snapshot.sh create mode 100755 plugins/vibe/skills/codex-fleet-monitor/scripts/watch.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index d90247a9..fe9de9e7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,6 +26,11 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). #### Fixed - The helper's `/health` and old token endpoint no longer reveal the user's home path to other websites: `/health` shares its details only with file pages and pages from this machine, and the token endpoint's errors name no paths. +### vibe v1.0.57 + +#### Added +- `codex-fleet-monitor` skill: a background event watcher for a fleet of Codex App Server threads (local and one ssh host) plus the pull requests they produce. It exits with a diff on the first change and flags IDLE, STALLED, FAILED, merge conflicts, review and comment counts, check changes and new PRs. Liveness is read from each thread's last turn and item count rather than the turn id alone, so a turn left open by a silent Codex usage limit is reported as STALLED. + ### vibe v1.0.56 #### Changed diff --git a/plugins/vibe/.claude-plugin/plugin.json b/plugins/vibe/.claude-plugin/plugin.json index beebf8fd..1b5c3142 100644 --- a/plugins/vibe/.claude-plugin/plugin.json +++ b/plugins/vibe/.claude-plugin/plugin.json @@ -1,7 +1,7 @@ { "name": "vibe", "description": "Vibe sessions, handoff, seed refresh, and ClosedLoop ticket workflows", - "version": "1.0.56", + "version": "1.0.57", "dependencies": ["closedloop-core"], "author": {"name": "ClosedLoop", "email": "support@closedloop.ai"} } diff --git a/plugins/vibe/.codex-plugin/plugin.json b/plugins/vibe/.codex-plugin/plugin.json index 6cca7ade..cb600603 100644 --- a/plugins/vibe/.codex-plugin/plugin.json +++ b/plugins/vibe/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "vibe", - "version": "1.0.56", + "version": "1.0.57", "description": "Vibe sessions, handoff, seed refresh, and ClosedLoop ticket workflows", "author": {"name": "ClosedLoop", "email": "support@closedloop.ai"}, "homepage": "https://github.com/closedloop-ai/claude-plugins", diff --git a/plugins/vibe/README.md b/plugins/vibe/README.md index b8121aa3..4478f140 100644 --- a/plugins/vibe/README.md +++ b/plugins/vibe/README.md @@ -301,6 +301,16 @@ through the queue, and mark the ticket Done. The fix worker runs the full web and Desktop screen walk itself, on a Mac through `pnpm vibe walk-linux`, and its counts and result file go into the PR's Test plan and the ticket's loop. +### `codex-fleet-monitor` + +Background event watcher for a fleet of long-running Codex App Server threads +(local and one ssh host) plus the pull requests they produce. One command polls +every watched thread and PR and exits with a diff on the first change: a session +going idle, stalling, or failing, a merge conflict, new review or comment +counts, changed checks, or a new PR. Liveness is read from the thread's last +turn and item count, so a turn left open by a silent Codex usage limit shows as +`STALLED` instead of running forever. + ### ClosedLoop Ticket Skills The plugin bundles the ClosedLoop ticket automation skill pack: `cl-policy`, `cl-analyze`, `cl-find-related-tickets`, `cl-split`, `cl-work-report`, `cl-sweep`, and `cl-execute`. Together they cover policy-aware ticket analysis, related-ticket discovery, safe one-level split workflows, private project work reports, sweep coordination, and end-to-end ticket execution. The imported sweep scripts use the native managed Codex App Server through bundled helpers rather than depending on a separate `app-server-orchestrator` skill. Runtime access still depends on the operator's normal ClosedLoop, GitHub, `workflow-memory`, and `closedloop-graph` MCP/API configuration. diff --git a/plugins/vibe/skills/codex-fleet-monitor/SKILL.md b/plugins/vibe/skills/codex-fleet-monitor/SKILL.md new file mode 100644 index 00000000..a0317e2d --- /dev/null +++ b/plugins/vibe/skills/codex-fleet-monitor/SKILL.md @@ -0,0 +1,68 @@ +--- +name: codex-fleet-monitor +description: Event watcher for a fleet of long-running Codex app-server threads (on this machine and/or one ssh host) plus the GitHub PRs they produce. Use when orchestrating several Codex sessions at once and you need to be woken on anything that needs attention - a session going idle, stalling (for example a silent usage limit), failing, a PR getting new comments or reviews, checks changing, a merge conflict, or a new PR opening. Runs as one background command that exits with a diff on the first change. +--- + +# codex-fleet-monitor + +One background command that polls every watched Codex thread and PR, and exits with a diff the moment any line changes. You read the diff, act, then re-arm it. It costs no model tokens while waiting. + +## Why liveness is read from the thread, not the turn id + +A Codex turn that hits a usage limit stays open and simply stops producing items. Checking only "is a turn active" reports it as running forever. This monitor reads each thread's last turn (status, error, item count) and flags a turn whose item count has not grown for `STALL_SECONDS` as `STALLED`. + +## Requirements + +- Each Codex session has a state file `.json` containing `threadId`, as written by the `app-server-orchestrator` skill. `scripts/liveness.mjs` imports that skill's `app-server-client.mjs` from `~/.codex/skills/app-server-orchestrator/scripts/`; set `APP_SERVER_CLIENT` to its path if it lives elsewhere. +- `codex`, `node`, `jq`, `gh` (authenticated) on this machine; `codex` and `node` on the remote host, reachable through `REMOTE_SHELL`. + +## Set up a work dir + +Pick a scratch dir (one per orchestration) and create: + +`config.env` + +```sh +LOCAL_STATE_DIR=/path/to/local/orchestrator/state/files +REMOTE_HOST=user@host # omit when there are no remote sessions +REMOTE_STATE_DIR=/path/on/remote # state files on the remote host +REMOTE_SHELL="zsh -lic" # login shell so node/codex are on PATH (default) +STALL_SECONDS=720 # no new thread items for this long means STALLED (default 12m) +WATCH_NEW_PR_REPOS="owner/repo" # optional: list open PRs here to spot new ones +PR_SEARCH="head:feature/" # optional: search filter for that list +BOT_LOGIN_REGEX="vercel|codex|github-actions|copilot|bot" # authors not counted as human feedback +# MY_GH_LOGIN=me # your own comments are not counted; defaults to `gh api user` +``` + +`sessions.txt`: one line per Codex session, `local|remote `. + +`prs.txt`: one line per PR to watch, `owner/repo `. Add a PR when it opens, remove it when it merges. + +## Run + +```sh +scripts/watch.sh [intervalSeconds] +``` + +Start it with `run_in_background`. You are notified when it exits. Read its output, handle every line, then start it again; it only watches while running. Editing `sessions.txt` or `prs.txt` takes effect on the next re-arm. `scripts/snapshot.sh ` prints the current state once. + +## Reading the output + +| Line | Meaning | Usual action | +|---|---|---| +| `session RUN` | turn in progress and producing items | none | +| `session ... IDLE` | turn completed; the session is waiting for you | read its last message, give it the next turn | +| `session ... STALLED(...)` | turn open but silent | check for a usage limit or a hung command; tell the user if it needs them | +| `session ... FAILED(LIMIT)` | turn errored on usage or rate limit | tell the user right away; a reset or second account is theirs to do | +| `session ... FAILED(ERROR)` / `STOPPED(...)` / `READFAIL` | turn errored, was interrupted, or the state file is unreadable | inspect the thread; restart or resume | +| `session remote UNREACHABLE` | ssh failed | check the host | +| `open-prs ` | open PR list changed | add the new PR to `prs.txt` and start its review | +| `pr ... comments= reviews= inline= checks=` | any field changed | new head: re-review; count up: read the feedback; `CONFLICTING`: tell the owning session to merge the base branch; checks: act on failures | + +Surface everything that needs a human to the user immediately: limits, stalls, failures, and new human review comments. + +## Notes + +- `mergeable` keeps its last known value while GitHub reports `UNKNOWN`, so it does not flap. +- Item history lives in `/item-history.txt`; delete it to reset stall timers. +- Don't run two copies on the same work dir. diff --git a/plugins/vibe/skills/codex-fleet-monitor/agents/openai.yaml b/plugins/vibe/skills/codex-fleet-monitor/agents/openai.yaml new file mode 100644 index 00000000..43678c69 --- /dev/null +++ b/plugins/vibe/skills/codex-fleet-monitor/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Codex Fleet Monitor" + short_description: "Get woken when a Codex session or PR needs attention" + default_prompt: "Use $codex-fleet-monitor to watch my Codex sessions and their pull requests." diff --git a/plugins/vibe/skills/codex-fleet-monitor/scripts/classify.sh b/plugins/vibe/skills/codex-fleet-monitor/scripts/classify.sh new file mode 100755 index 00000000..e6c7a13e --- /dev/null +++ b/plugins/vibe/skills/codex-fleet-monitor/scripts/classify.sh @@ -0,0 +1,16 @@ +#!/bin/zsh +# stdin: " ". $1: item history file. $2: stall seconds. +# A running turn whose item count has not grown for $2 seconds is STALLED (a Codex usage limit +# leaves the turn open and silent, so "turn is open" alone is not liveness). +H=$1; STALL=${2:-720}; touch $H; now=$(date +%s) +while read where n st items flag; do + prev=$(grep "^$where:$n " $H | awk '{print $2, $3}'); pi=${prev% *}; pt=${prev#* } + if [ -z "$prev" ] || [ "$pi" != "$items" ]; then pt=$now; fi + grep -v "^$where:$n " $H > $H.tmp; echo "$where:$n $items $pt" >> $H.tmp; mv $H.tmp $H + if [ "$flag" != "-" ]; then s="FAILED($flag)" + elif [ "$st" = inProgress ] && [ $((now - pt)) -ge $STALL ]; then s="STALLED(no progress $((STALL / 60))m+)" + elif [ "$st" = inProgress ]; then s=RUN + elif [ "$st" = completed ]; then s=IDLE + else s="STOPPED($st)"; fi + echo "session $where $n $s" +done diff --git a/plugins/vibe/skills/codex-fleet-monitor/scripts/liveness.mjs b/plugins/vibe/skills/codex-fleet-monitor/scripts/liveness.mjs new file mode 100644 index 00000000..ea09fa84 --- /dev/null +++ b/plugins/vibe/skills/codex-fleet-monitor/scripts/liveness.mjs @@ -0,0 +1,37 @@ +// Usage: node liveness.mjs ... +// Reads /.json (an app-server-orchestrator state file with a threadId) +// and prints " "; flag is LIMIT, ERROR, READFAIL or -. +import { execFileSync } from 'node:child_process'; +import { readFileSync } from 'node:fs'; +import { homedir } from 'node:os'; + +setTimeout(() => process.exit(1), 120_000).unref(); + +const clientPath = + process.env.APP_SERVER_CLIENT ?? + `${homedir()}/.codex/skills/app-server-orchestrator/scripts/app-server-client.mjs`; +const { AppServerClient, initializeClient } = await import(clientPath); +const [dir, ...names] = process.argv.slice(2); +let socket = `${homedir()}/.codex/app-server-control/app-server-control.sock`; +try { + const started = execFileSync('codex', ['app-server', 'daemon', 'start'], { encoding: 'utf8' }); + socket = JSON.parse(started).socketPath ?? socket; +} catch {} +const client = new AppServerClient({ transport: 'proxy', socket }); +await initializeClient(client, { name: 'fleet-liveness', title: 'Fleet liveness', version: '1.0.0' }); +for (const n of names) { + try { + const { threadId } = JSON.parse(readFileSync(`${dir}/${n}.json`, 'utf8')); + const r = await client.request('thread/read', { threadId, includeTurns: true }); + const t = (r.thread?.turns ?? []).at(-1); + const err = JSON.stringify(t?.error ?? '').toLowerCase(); + let flag = '-'; + if (/usage|limit|quota|rate/.test(err)) flag = 'LIMIT'; + else if (t?.error) flag = 'ERROR'; + console.log(`${n} ${t?.status ?? 'none'} ${(t?.items ?? []).length} ${flag}`); + } catch { + console.log(`${n} unreadable 0 READFAIL`); + } +} +client.close(); +process.exit(0); diff --git a/plugins/vibe/skills/codex-fleet-monitor/scripts/snapshot.sh b/plugins/vibe/skills/codex-fleet-monitor/scripts/snapshot.sh new file mode 100755 index 00000000..2a3825c5 --- /dev/null +++ b/plugins/vibe/skills/codex-fleet-monitor/scripts/snapshot.sh @@ -0,0 +1,37 @@ +#!/bin/zsh +# Usage: snapshot.sh . One line per watched item; any changed line is an event. +K=${0:A:h}; W=${1:?workDir}; [ -f $W/config.env ] && source $W/config.env +: ${STALL_SECONDS:=720} ${BOT_LOGIN_REGEX:=vercel|codex|github-actions|copilot|bot} ${REMOTE_SHELL:=zsh -lic} +: > $W/live.txt +L=$(awk '$1=="local"{print $2}' $W/sessions.txt 2>/dev/null | tr '\n' ' ') +if [ -n "${L// }" ]; then + node $K/liveness.mjs ${LOCAL_STATE_DIR:?set LOCAL_STATE_DIR} ${=L} | sed 's/^/local /' >> $W/live.txt +fi +R=$(awk '$1=="remote"{print $2}' $W/sessions.txt 2>/dev/null | tr '\n' ' ') +if [ -n "${R// }" ]; then + rl=$REMOTE_STATE_DIR/.fleet-liveness.mjs + out=$(ssh -o ConnectTimeout=10 -o ServerAliveInterval=15 ${REMOTE_HOST:?set REMOTE_HOST} \ + "cat > $rl && $REMOTE_SHELL 'node $rl $REMOTE_STATE_DIR $R'" < $K/liveness.mjs 2>/dev/null) + if [ -n "$out" ]; then echo "$out" | sed 's/^/remote /' >> $W/live.txt + else echo "session remote $REMOTE_HOST UNREACHABLE"; fi +fi +zsh $K/classify.sh $W/item-history.txt $STALL_SECONDS < $W/live.txt +for r in ${=WATCH_NEW_PR_REPOS}; do + echo "open-prs $r $(gh pr list -R $r --state open --search "${PR_SEARCH:-}" --json number --jq '.[].number' 2>/dev/null | sort -n | tr '\n' ' ')" +done +[ -s $W/prs.txt ] || exit 0 +ME=${MY_GH_LOGIN:-$(gh api user --jq .login 2>/dev/null)} +skip="(.user.login==\"$ME\" or (.user.login|test(\"$BOT_LOGIN_REGEX\";\"i\")))" +while read r n; do + [ -z "$n" ] && continue + st=$(gh pr view $n -R $r --json state,headRefOid,reviewDecision,mergeable --jq '"\(.state) \(.headRefOid[0:8]) \(.reviewDecision) \(.mergeable)"' 2>/dev/null) + m=${st##* }; k=$W/merge-${r//\//_}-$n.txt + # GitHub reports UNKNOWN while it recomputes; keep the last known value so it does not flap. + if [ "$m" = UNKNOWN ] && [ -f $k ]; then m=$(cat $k); else echo $m > $k; fi + st="${st% *} $m" + c=$(gh api "repos/$r/issues/$n/comments" --jq "[.[]|select($skip|not)]|length" 2>/dev/null) + v=$(gh api "repos/$r/pulls/$n/reviews" --jq "[.[]|select($skip|not)]|length" 2>/dev/null) + i=$(gh api "repos/$r/pulls/$n/comments" --jq "[.[]|select($skip|not)]|length" 2>/dev/null) + ck=$(gh pr checks $n -R $r 2>/dev/null | awk -F'\t' '{print $2}' | sort | uniq -c | tr -s ' ' | tr '\n' ',') + echo "pr $r#$n $st comments=$c reviews=$v inline=$i checks=$ck" +done < $W/prs.txt diff --git a/plugins/vibe/skills/codex-fleet-monitor/scripts/watch.sh b/plugins/vibe/skills/codex-fleet-monitor/scripts/watch.sh new file mode 100755 index 00000000..e0f94331 --- /dev/null +++ b/plugins/vibe/skills/codex-fleet-monitor/scripts/watch.sh @@ -0,0 +1,12 @@ +#!/bin/zsh +# Usage: watch.sh [intervalSeconds]. Exits with the diff on the first change. +K=${0:A:h}; W=${1:?workDir}; I=${2:-120} +base=$(zsh $K/snapshot.sh $W); echo "$base" > $W/last-snapshot.txt +for _ in $(seq 1 200); do + sleep $I + now=$(zsh $K/snapshot.sh $W) + if [ -n "$now" ] && [ "$now" != "$base" ]; then + echo "$now" > $W/last-snapshot.txt; echo EVENT; diff <(echo "$base") <(echo "$now") | grep '^[<>]'; exit 0 + fi +done +echo TIMEOUT; cat $W/last-snapshot.txt From 88fcf98541f0dfd49a982a5ee3c82754810bfceb Mon Sep 17 00:00:00 2001 From: Daniel Ochoa Date: Mon, 19 Oct 2026 16:44:03 -0500 Subject: [PATCH 2/5] Rewrite codex-fleet-monitor skill description - Describe when to reach for the skill and why a turn-only watcher misses a usage-limit stall --- plugins/vibe/skills/codex-fleet-monitor/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/vibe/skills/codex-fleet-monitor/SKILL.md b/plugins/vibe/skills/codex-fleet-monitor/SKILL.md index a0317e2d..a3125f4b 100644 --- a/plugins/vibe/skills/codex-fleet-monitor/SKILL.md +++ b/plugins/vibe/skills/codex-fleet-monitor/SKILL.md @@ -1,6 +1,6 @@ --- name: codex-fleet-monitor -description: Event watcher for a fleet of long-running Codex app-server threads (on this machine and/or one ssh host) plus the GitHub PRs they produce. Use when orchestrating several Codex sessions at once and you need to be woken on anything that needs attention - a session going idle, stalling (for example a silent usage limit), failing, a PR getting new comments or reviews, checks changing, a merge conflict, or a new PR opening. Runs as one background command that exits with a diff on the first change. +description: 'Use PROACTIVELY whenever you start or are already running two or more Codex or vibe sessions (via app-server-orchestrator), or are watching PRs on their behalf, and need to be told when anything needs attention. Replaces hand-rolled polling loops: a Codex turn that hits a usage limit stays "active" and silent, so a watcher that checks only for an open turn misses it. This reads each thread''s progress and flags IDLE, STALLED, FAILED(LIMIT), new PR comments or reviews, check changes, merge conflicts, and new PRs.' --- # codex-fleet-monitor From 375507d945fece4a5b08e536fedf08298f46b586 Mon Sep 17 00:00:00 2001 From: Daniel Ochoa Date: Mon, 19 Oct 2026 16:44:14 -0500 Subject: [PATCH 3/5] Note skill trigger in changelog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index fe9de9e7..fb092716 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,7 +29,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### vibe v1.0.57 #### Added -- `codex-fleet-monitor` skill: a background event watcher for a fleet of Codex App Server threads (local and one ssh host) plus the pull requests they produce. It exits with a diff on the first change and flags IDLE, STALLED, FAILED, merge conflicts, review and comment counts, check changes and new PRs. Liveness is read from each thread's last turn and item count rather than the turn id alone, so a turn left open by a silent Codex usage limit is reported as STALLED. +- `codex-fleet-monitor` skill: a background event watcher for a fleet of Codex App Server threads (local and one ssh host) plus the pull requests they produce. It exits with a diff on the first change and flags IDLE, STALLED, FAILED, merge conflicts, review and comment counts, check changes and new PRs. Liveness is read from each thread's last turn and item count rather than the turn id alone, so a turn left open by a silent Codex usage limit is reported as STALLED. The skill description tells agents to reach for it whenever two or more Codex or vibe sessions are running. ### vibe v1.0.56 From 9ba25548b5ba382742ecc3223e20940288f19412 Mon Sep 17 00:00:00 2001 From: Daniel Ochoa Date: Mon, 19 Oct 2026 16:58:58 -0500 Subject: [PATCH 4/5] Make codex-fleet-monitor self-contained and accept thread ids liveness.mjs no longer imports the app-server-orchestrator skill from ~/.codex/skills. It carries a small App Server client (WebSocket over codex app-server proxy, initialize, thread/read) in one file so the ssh copy to a remote host still works. sessions.txt takes 'local|remote