From 0770ff287fbff84bafa61279fd300ea5f667850f Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 07:23:21 -0700 Subject: [PATCH 01/64] =?UTF-8?q?docs(runbook):=20managed-runtime=20redepl?= =?UTF-8?q?oy=20section=20=E2=80=94=20build=20surface,=20feature=20flag,?= =?UTF-8?q?=20unit=20coupling,=20restart=20effects,=20verification?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/development/managed-runtime.md | 7 ++ .../development/self-hosted-launch-runbook.md | 76 +++++++++++++++++++ 2 files changed, 83 insertions(+) diff --git a/docs/development/managed-runtime.md b/docs/development/managed-runtime.md index 3e0e459ce..5609b46c9 100644 --- a/docs/development/managed-runtime.md +++ b/docs/development/managed-runtime.md @@ -8,6 +8,13 @@ server built with `managed-runtime-v1`, booted with the runtime controller, and a client that negotiated `managedRuntimeV1` can use it. Other routes remain legacy. +Deploying or restarting the managed runtime is an operator procedure with its +own build surface, unit coupling, and restart effects — see "Redeploy managed +runtime" in the +[self-hosted launch runbook](self-hosted-launch-runbook.md). The build must +compile the feature in (`--features freshell-server/managed-runtime-v1`); the +runtime env var alone arms nothing. + ## Current release qualification OpenCode 1.18.21 (`openai/gpt-5.6-luna`, using the configured OpenAI OAuth diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index f78e29c64..1ced9581d 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -74,6 +74,82 @@ re-run is the ratified tradeoff, not a gap — the failing provider's error names the exact path. The session host's `operator_home_readiness` event confirms the home root's group traversal afterward. +## Redeploy managed runtime (build, units, restart effects) + +The managed runtime spans THREE binaries, one opt-in feature flag, and two +systemd user units. A server-only rebuild or restart is NOT enough to change +soul behavior — and a plain rebuild silently loses it. + +**Build surface.** Build all three from the repo root when any of them +changes: + +```bash +cargo build --release -p freshell-server -p freshell-supervisor \ + -p freshell-session-host --features freshell-server/managed-runtime-v1 +``` + +- The server's soul controller is strictly opt-in (`default = []` in + `crates/freshell-server/Cargo.toml`). Setting `FRESHELL_MANAGED_RUNTIME_V1=1` + in the unit enables nothing by itself — the env var only arms a binary that + was compiled with the feature. A server built without it serves fresh panes + on the HOST (no soul isolation) and answers `404` on `/api/runtime/*`. +- The SUPERVISOR creates and owns soul containers (`freshell-supervisor + serve ...`); the home mount, masks, and the `--operator-home` flag live + there. It must match the server's expectations, so redeploy it together + with the server. +- The SESSION HOST binary is injected into every NEW soul at container + creation (the supervisor's `--host-binary` argument). New souls pick the + new binary automatically; running souls keep the one they were created + with until their next incarnation. + +**Units and restart coupling.** With the systemd topology +(`installers/systemd/freshell-rust.service` + its `managed-runtime.conf` +drop-in, plus `freshell-supervisor.service`): + +- `freshell-rust.service` has `Requires=` + `After=` on + `freshell-supervisor.service`. Restarting the SUPERVISOR therefore also + restarts the server in the same transaction — one + `systemctl --user restart freshell-supervisor.service` redeploys both. +- Restarting the server alone (`systemctl --user restart + freshell-rust.service`) leaves the supervisor running. Use this for + server-only changes. +- Restarting the production server always requires the user's explicit + approval (`APPROVED`), including the coupled bounce above. + +**What a restart does:** + +- **Existing souls keep running.** Soul containers are owned by the Docker + daemon and tracked by the supervisor's durable registry, not by the server + process. After the restart, soul-backed panes reattach on the next client + connect; the supervisor re-verifies each running container against its + persisted creation-time state, so pre-restart souls pass with no + migration. Souls keep the mounts they were created with (running souls + gain the operator-home mount only at their next incarnation). +- **Non-soul terminals end.** Host terminals (shell panes and host-run CLI + panes) are child processes of the server, so a restart terminates them. + Their panes rehydrate as exited terminals; resume those conversations + through the CLI's own durable storage (`claude --resume`, + `codex resume`, opencode's session picker), not through Freshell. + +**Post-deploy verification checklist:** + +1. Health: `curl -fsS http://127.0.0.1:3001/api/health` → `"ready": true`. +2. Feature present: with the auth token, + `GET /api/runtime/readiness` answers `200`. A `404` means the running + binary was built WITHOUT the feature — rebuild with the command above + and restart again. +3. Units: `systemctl --user is-active freshell-rust.service + freshell-supervisor.service` → both `active`. +4. Existing souls: `docker ps --filter name=incarnation` shows the same + containers with unchanged uptimes. +5. New-soul probe: create a terminal pane in a real workspace and confirm a + NEW `freshell-incarnation-*` container appears (`docker ps`), that it + carries the operator-home bind and the + `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that + its session host emitted one `operator_home_readiness` event + (`docker exec cat /run/freshell/host-lifecycle.jsonl`). Close + the probe pane afterward. + ## Launch the native Windows client Run this in a native Windows PowerShell terminal: From 81c0244009f15d2de8b48f63582b162f807698ec Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:08:56 -0700 Subject: [PATCH 02/64] =?UTF-8?q?docs(runbook):=20fold=20restart-test=20fi?= =?UTF-8?q?ndings=20=E2=80=94=20reconnect=20soul=20creation=20is=20reattac?= =?UTF-8?q?h=20not=20replacement;=20probe=20soul=20reaped=20on=20idle=20ti?= =?UTF-8?q?meout?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/development/self-hosted-launch-runbook.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 1ced9581d..8c2d31a85 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -141,14 +141,18 @@ drop-in, plus `freshell-supervisor.service`): 3. Units: `systemctl --user is-active freshell-rust.service freshell-supervisor.service` → both `active`. 4. Existing souls: `docker ps --filter name=incarnation` shows the same - containers with unchanged uptimes. + containers with unchanged uptimes. A NEW container may appear within + seconds of the restart — that is a client reconnecting and a durable + terminal pane reattaching, not an old soul being replaced; the + pre-restart containers must keep their uptimes. 5. New-soul probe: create a terminal pane in a real workspace and confirm a NEW `freshell-incarnation-*` container appears (`docker ps`), that it carries the operator-home bind and the `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that its session host emitted one `operator_home_readiness` event (`docker exec cat /run/freshell/host-lifecycle.jsonl`). Close - the probe pane afterward. + the probe pane afterward; its container is collected by the idle + reaper after the idle timeout, not immediately on close. ## Launch the native Windows client From caa37d392ee6a10c1e4b7cb85741e565d72c408c Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:19:06 -0700 Subject: [PATCH 03/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=201=20?= =?UTF-8?q?=E2=80=94=20shells=20are=20managed=20souls;=20host-side=20termi?= =?UTF-8?q?nal=20scope;=20unit=20port=20note;=20workload=20image=20surface?= =?UTF-8?q?;=20probe=20soul=20is=20not=20auto-reaped?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 50 ++++++++++++++----- 1 file changed, 37 insertions(+), 13 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 8c2d31a85..a7208e3a4 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -76,12 +76,12 @@ confirms the home root's group traversal afterward. ## Redeploy managed runtime (build, units, restart effects) -The managed runtime spans THREE binaries, one opt-in feature flag, and two -systemd user units. A server-only rebuild or restart is NOT enough to change -soul behavior — and a plain rebuild silently loses it. +The managed runtime spans three binaries, a pinned workload image, one opt-in +feature flag, and two systemd user units. A server-only rebuild or restart is +NOT enough to change soul behavior — and a plain rebuild silently loses it. -**Build surface.** Build all three from the repo root when any of them -changes: +**Build surface.** Build all three binaries from the repo root when any of +them changes: ```bash cargo build --release -p freshell-server -p freshell-supervisor \ @@ -101,6 +101,16 @@ cargo build --release -p freshell-server -p freshell-supervisor \ creation (the supervisor's `--host-binary` argument). New souls pick the new binary automatically; running souls keep the one they were created with until their next incarnation. +- The WORKLOAD IMAGE is a fourth deploy surface: every soul runs the image + recorded in `~/.config/freshell/managed-runtime.env` as + `FRESHELL_RUNTIME_IMAGE` (an immutable `sha256:` ID — build it from + `docker/runtime/Dockerfile` per + [managed-runtime-rollout.md](managed-runtime-rollout.md)). Rebuilding the + Rust binaries never updates it. To deploy image changes: build the new + image, record its `sha256:` ID in that file, and restart + `freshell-supervisor.service` (the unit reads the file at start). New + souls use the new image; running souls keep theirs until their next + incarnation. **Units and restart coupling.** With the systemd topology (`installers/systemd/freshell-rust.service` + its `managed-runtime.conf` @@ -115,6 +125,10 @@ drop-in, plus `freshell-supervisor.service`): server-only changes. - Restarting the production server always requires the user's explicit approval (`APPROVED`), including the coupled bounce above. +- The deployed unit's port wins: the checked-in installer unit defaults to + `PORT=3002`, while this host's installed unit serves `PORT=3001`. The + commands below use 3001 — substitute the port the deployed unit actually + sets. **What a restart does:** @@ -124,12 +138,17 @@ drop-in, plus `freshell-supervisor.service`): connect; the supervisor re-verifies each running container against its persisted creation-time state, so pre-restart souls pass with no migration. Souls keep the mounts they were created with (running souls - gain the operator-home mount only at their next incarnation). -- **Non-soul terminals end.** Host terminals (shell panes and host-run CLI - panes) are child processes of the server, so a restart terminates them. - Their panes rehydrate as exited terminals; resume those conversations - through the CLI's own durable storage (`claude --resume`, - `codex resume`, opencode's session picker), not through Freshell. + gain the operator-home mount only at their next incarnation). With the + managed runtime live this includes SHELL panes — `shell` mode routes + through souls, so shells survive the restart and reattach like any other + soul-backed pane. +- **Host-side terminals end.** Only terminals the server hosts as its own + child processes die with it: panes created while the feature was missing + (they silently spawned host-side), panes from clients that did not + negotiate `managedRuntimeV1`, and host-run CLI panes. Their panes + rehydrate as exited terminals; resume those conversations through the + CLI's own durable storage (`claude --resume`, `codex resume`, opencode's + session picker), not through Freshell. **Post-deploy verification checklist:** @@ -151,8 +170,13 @@ drop-in, plus `freshell-supervisor.service`): `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that its session host emitted one `operator_home_readiness` event (`docker exec cat /run/freshell/host-lifecycle.jsonl`). Close - the probe pane afterward; its container is collected by the idle - reaper after the idle timeout, not immediately on close. + the probe pane afterward. The probe soul is NOT auto-collected: the + server's idle reaper explicitly skips managed terminals (their lifetime + is supervisor-owned) and the supervisor has no idle reaper, so the + container persists as a durable background session — the same lifecycle + as any closed pane's soul. To remove it immediately, `docker stop` the + container; a later reattach mints a new incarnation rather than resuming + it. ## Launch the native Windows client From 31febee9dc6ce7e17b8528646b8ef853c0e91352 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:24:39 -0700 Subject: [PATCH 04/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=202=20?= =?UTF-8?q?=E2=80=94=20fenced=20API=20for=20probe=20cleanup=20(external=20?= =?UTF-8?q?docker=20stop=20trips=20supervisor=20recovery);=20launch-rust.s?= =?UTF-8?q?h=20recovery=20caveat=20on=20managed=20instances;=20assert=20re?= =?UTF-8?q?adiness=20event=20values;=20artifact=20freshness=20check;=20pro?= =?UTF-8?q?be=20workspace=20must=20not=20equal=20operator=20home?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 47 ++++++++++++++----- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index a7208e3a4..defa96ead 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -57,6 +57,13 @@ launches the Rust binary directly — its `ExecStart` is the no dependency preparation; prepare the checkout once on that machine as above, and the service restores the server after restarts. +On a managed-runtime instance (systemd topology, see "Redeploy managed +runtime" below), do NOT recover a dead server with +`scripts/launch-rust.sh`: it rebuilds `freshell-server` WITHOUT the +`managed-runtime-v1` feature and would overwrite the feature-enabled +binary, silently reverting the instance to host-side terminals. Use the +redeploy procedure in that section instead. + ## Prepare the operator home (managed-runtime souls) Managed-runtime souls mount the operator home read-only so config `{file:}` @@ -164,19 +171,37 @@ drop-in, plus `freshell-supervisor.service`): seconds of the restart — that is a client reconnecting and a durable terminal pane reattaching, not an old soul being replaced; the pre-restart containers must keep their uptimes. -5. New-soul probe: create a terminal pane in a real workspace and confirm a - NEW `freshell-incarnation-*` container appears (`docker ps`), that it +5. New-soul probe: create a terminal pane in a real workspace that is NOT + the operator home itself (a subdirectory of it is fine) — when the + workspace equals the operator home, the supervisor deliberately omits + the separate home bind (the workspace bind already provides it), so the + checks below do not apply to that launch. Confirm a NEW + `freshell-incarnation-*` container appears (`docker ps`), that it carries the operator-home bind and the `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that - its session host emitted one `operator_home_readiness` event - (`docker exec cat /run/freshell/host-lifecycle.jsonl`). Close - the probe pane afterward. The probe soul is NOT auto-collected: the - server's idle reaper explicitly skips managed terminals (their lifetime - is supervisor-owned) and the supervisor has no idle reaper, so the - container persists as a durable background session — the same lifecycle - as any closed pane's soul. To remove it immediately, `docker stop` the - container; a later reattach mints a new incarnation rather than resuming - it. + its session host emitted an `operator_home_readiness` event + (`docker exec cat /run/freshell/host-lifecycle.jsonl`) + whose payload reports `"present":true` and + `"home_root_traversable_by_child":true` — the event is emitted with its + findings even when the home is missing or unreadable, so its mere + presence is NOT proof of readiness. Close the probe pane afterward. +6. Artifact freshness (after a binary or image redeploy): the new probe + container's image (`docker inspect --format '{{.Image}}'`) + equals the `sha256:` ID recorded in `~/.config/freshell/managed-runtime.env`, + and both units' `ExecMainStartTimestamp` postdates the artifacts you just + built. +7. Probe cleanup: the probe soul is NOT auto-collected — the server's + idle reaper explicitly skips managed terminals (their lifetime is + supervisor-owned) and the supervisor has no idle reaper, so the + container persists as a durable background session. To remove it + immediately, stop the soul through the fenced runtime API: + `POST /api/runtime/souls//stop` (find the soul ID with + `GET /api/runtime/souls`, or read the container's `com.freshell.soul-id` + docker label). Do NOT `docker stop` the container directly — the + supervisor treats an externally stopped container as an unexpected exit + and runs its automatic recovery path, which may recreate the soul or + mark a session without a durable resume path as lost. Leaving the probe + soul running is also fine; it is inert. ## Launch the native Windows client From 18bf3f87578862cb0f27000b186c67100731e702 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:31:07 -0700 Subject: [PATCH 05/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=203=20?= =?UTF-8?q?=E2=80=94=20full=20intent-revision-guarded=20stop=20request;=20?= =?UTF-8?q?capture=20pre-restart=20soul=20inventory=20for=20the=20comparis?= =?UTF-8?q?on?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 42 +++++++++++++------ 1 file changed, 30 insertions(+), 12 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index defa96ead..2a896b084 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -136,6 +136,9 @@ drop-in, plus `freshell-supervisor.service`): `PORT=3002`, while this host's installed unit serves `PORT=3001`. The commands below use 3001 — substitute the port the deployed unit actually sets. +- Before restarting, capture the soul inventory so the post-deploy checks + can prove nothing was replaced: + `docker ps --filter name=incarnation --format '{{.Names}}\t{{.Status}}' | tee /tmp/souls-before.txt`. **What a restart does:** @@ -167,10 +170,11 @@ drop-in, plus `freshell-supervisor.service`): 3. Units: `systemctl --user is-active freshell-rust.service freshell-supervisor.service` → both `active`. 4. Existing souls: `docker ps --filter name=incarnation` shows the same - containers with unchanged uptimes. A NEW container may appear within - seconds of the restart — that is a client reconnecting and a durable - terminal pane reattaching, not an old soul being replaced; the - pre-restart containers must keep their uptimes. + containers as the pre-restart capture (step above) with unchanged + uptimes — a container missing from the capture was lost, and a NEW + container may appear within seconds of the restart: that is a client + reconnecting and a durable terminal pane reattaching, not an old soul + being replaced; the pre-restart containers must keep their uptimes. 5. New-soul probe: create a terminal pane in a real workspace that is NOT the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits @@ -194,14 +198,28 @@ drop-in, plus `freshell-supervisor.service`): idle reaper explicitly skips managed terminals (their lifetime is supervisor-owned) and the supervisor has no idle reaper, so the container persists as a durable background session. To remove it - immediately, stop the soul through the fenced runtime API: - `POST /api/runtime/souls//stop` (find the soul ID with - `GET /api/runtime/souls`, or read the container's `com.freshell.soul-id` - docker label). Do NOT `docker stop` the container directly — the - supervisor treats an externally stopped container as an unexpected exit - and runs its automatic recovery path, which may recreate the soul or - mark a session without a durable resume path as lost. Leaving the probe - soul running is also fine; it is inert. + immediately, stop the soul through the fenced runtime API (find the + soul ID with `GET /api/runtime/souls`, or read the container's + `com.freshell.soul-id` docker label), then send the intent-revision- + guarded stop with the auth token: + + ```bash + SOUL_ID=$(docker inspect \ + --format '{{index .Config.Labels "com.freshell.soul-id"}}') + REV=$(curl -s -H "x-auth-token: $TOK" \ + http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') + curl -s -X POST -H "x-auth-token: $TOK" \ + -H "Content-Type: application/json" \ + -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$REV}" \ + http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID/stop + ``` + + Do NOT `docker stop` the container directly — the supervisor treats an + externally stopped container as an unexpected exit and runs its + automatic recovery path, which may recreate the soul or mark a session + without a durable resume path as lost. Leaving the probe soul running + is also fine; it is inert. ## Launch the native Windows client From 8bf38a453c6ff7fd0705afca5929e541b803ca3c Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:40:40 -0700 Subject: [PATCH 06/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=204=20?= =?UTF-8?q?=E2=80=94=20client=20auto-reconciles=20host-side=20panes=20(Fre?= =?UTF-8?q?sh/Respawn);=20per-surface=20artifact=20freshness;=20OneCLI-pri?= =?UTF-8?q?mary=20cleanup=20with=20defined=20token=20and=20verified=5Fempt?= =?UTF-8?q?y=20outcome=20assertion;=20compare=20container=20identities=20n?= =?UTF-8?q?ot=20uptimes?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 65 ++++++++++++------- 1 file changed, 43 insertions(+), 22 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 2a896b084..78e1c2c3a 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -138,7 +138,7 @@ drop-in, plus `freshell-supervisor.service`): sets. - Before restarting, capture the soul inventory so the post-deploy checks can prove nothing was replaced: - `docker ps --filter name=incarnation --format '{{.Names}}\t{{.Status}}' | tee /tmp/souls-before.txt`. + `docker ps --filter name=incarnation --format '{{.ID}}\t{{.Names}}' | tee /tmp/souls-before.txt`. **What a restart does:** @@ -152,13 +152,17 @@ drop-in, plus `freshell-supervisor.service`): managed runtime live this includes SHELL panes — `shell` mode routes through souls, so shells survive the restart and reattach like any other soul-backed pane. -- **Host-side terminals end.** Only terminals the server hosts as its own - child processes die with it: panes created while the feature was missing - (they silently spawned host-side), panes from clients that did not - negotiate `managedRuntimeV1`, and host-run CLI panes. Their panes - rehydrate as exited terminals; resume those conversations through the - CLI's own durable storage (`claude --resume`, `codex resume`, opencode's - session picker), not through Freshell. +- **Host-side terminals end — and the client reconciles them.** Only + terminals the server hosts as its own child processes die with it: panes + created while the feature was missing (they silently spawned host-side), + panes from clients that did not negotiate `managedRuntimeV1`, and host-run + CLI panes. The process and any unsaved in-flight state are gone, but the + panes do not sit as exited terminals: on reconnect the client's reconcile + handshake replaces a lost shell with a fresh terminal automatically + (shells are stateless by design) and respawns a CLI pane with a durable + session from the CLI's own storage (`claude --resume`, `codex resume`, + opencode's session picker are the manual fallbacks when there is nothing + durable to resume). **Post-deploy verification checklist:** @@ -169,12 +173,13 @@ drop-in, plus `freshell-supervisor.service`): and restart again. 3. Units: `systemctl --user is-active freshell-rust.service freshell-supervisor.service` → both `active`. -4. Existing souls: `docker ps --filter name=incarnation` shows the same - containers as the pre-restart capture (step above) with unchanged - uptimes — a container missing from the capture was lost, and a NEW - container may appear within seconds of the restart: that is a client - reconnecting and a durable terminal pane reattaching, not an old soul - being replaced; the pre-restart containers must keep their uptimes. +4. Existing souls: every container in the pre-restart capture is still + running with its original start time (`docker ps` with + `{{.Names}}\t{{.CreatedAt}}`-style output, or `State.StartedAt` via + `docker inspect`) — a captured container missing afterward was lost. A + container absent from the capture may appear within seconds of the + restart: that is a client reconnecting and a durable terminal pane + reattaching, not an old soul being replaced. 5. New-soul probe: create a terminal pane in a real workspace that is NOT the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits @@ -189,21 +194,33 @@ drop-in, plus `freshell-supervisor.service`): `"home_root_traversable_by_child":true` — the event is emitted with its findings even when the home is missing or unreadable, so its mere presence is NOT proof of readiness. Close the probe pane afterward. -6. Artifact freshness (after a binary or image redeploy): the new probe - container's image (`docker inspect --format '{{.Image}}'`) - equals the `sha256:` ID recorded in `~/.config/freshell/managed-runtime.env`, - and both units' `ExecMainStartTimestamp` postdates the artifacts you just - built. +6. Artifact freshness, per changed surface: the new probe container's image + (`docker inspect --format '{{.Image}}'`) equals the + `sha256:` ID recorded in `~/.config/freshell/managed-runtime.env`, and + every unit you restarted shows an `ExecMainStartTimestamp` + (`systemctl --user show -p ExecMainStartTimestamp`) that + postdates the artifact it was restarted to deploy. On a server-only + restart the supervisor's timestamp legitimately predates the new server + build — only compare the units that were actually bounced. 7. Probe cleanup: the probe soul is NOT auto-collected — the server's idle reaper explicitly skips managed terminals (their lifetime is supervisor-owned) and the supervisor has no idle reaper, so the container persists as a durable background session. To remove it immediately, stop the soul through the fenced runtime API (find the soul ID with `GET /api/runtime/souls`, or read the container's - `com.freshell.soul-id` docker label), then send the intent-revision- - guarded stop with the auth token: + `com.freshell.soul-id` docker label). Protected runtime API calls go + through OneCLI per the rollout procedure (its grant is scoped to the + server host and `/api/runtime/*` and injects `x-auth-token` — see + [managed-runtime-rollout.md](managed-runtime-rollout.md)); the direct + form below is the equivalent for an operator holding the checkout's + token. The stop is intent-revision-guarded, and HTTP 200 alone proves + nothing — the response's `outcome` must be `verified_empty` (other + outcomes such as `termination_unconfirmed` or `blocked_ownership` mean + the soul is NOT confirmed stopped; re-check `GET /api/runtime/souls/$SOUL_ID` + and retry with the new revision if needed): ```bash + TOK=$(grep -E '^AUTH_TOKEN=' /home/dan/code/freshell/.env | cut -d= -f2-) SOUL_ID=$(docker inspect \ --format '{{index .Config.Labels "com.freshell.soul-id"}}') REV=$(curl -s -H "x-auth-token: $TOK" \ @@ -212,7 +229,11 @@ drop-in, plus `freshell-supervisor.service`): curl -s -X POST -H "x-auth-token: $TOK" \ -H "Content-Type: application/json" \ -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$REV}" \ - http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID/stop + http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID/stop \ + | python3 -c 'import json,sys; +o=json.load(sys.stdin) +assert o.get("outcome")=="verified_empty", o +print("stopped:", o["outcome"])' ``` Do NOT `docker stop` the container directly — the supervisor treats an From adc94e3dd25393871ec2a8779e5c13465d2fa901 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:47:58 -0700 Subject: [PATCH 07/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=205=20?= =?UTF-8?q?=E2=80=94=20record=20and=20verify=20probe=20soul=20identity=20b?= =?UTF-8?q?efore=20the=20stop;=20StartedAt-based=20survival=20comparison;?= =?UTF-8?q?=20prove=20running=20binaries=20via=20/proc=20exe;=20client=20b?= =?UTF-8?q?uild=20surface;=20fresh-agent=20sidecars=20in=20restart=20effec?= =?UTF-8?q?ts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 131 ++++++++++++------ 1 file changed, 89 insertions(+), 42 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 78e1c2c3a..540d57fee 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -83,9 +83,10 @@ confirms the home root's group traversal afterward. ## Redeploy managed runtime (build, units, restart effects) -The managed runtime spans three binaries, a pinned workload image, one opt-in -feature flag, and two systemd user units. A server-only rebuild or restart is -NOT enough to change soul behavior — and a plain rebuild silently loses it. +The managed runtime spans three binaries, a pinned workload image, the web +client, one opt-in feature flag, and two systemd user units. A server-only +rebuild or restart is NOT enough to change soul behavior — and a plain +rebuild silently loses it. **Build surface.** Build all three binaries from the repo root when any of them changes: @@ -118,6 +119,12 @@ cargo build --release -p freshell-server -p freshell-supervisor \ `freshell-supervisor.service` (the unit reads the file at start). New souls use the new image; running souls keep theirs until their next incarnation. +- The WEB CLIENT is a fifth deploy surface: the server serves `dist/client` + from disk, and only a client that negotiates `managedRuntimeV1` can create + managed terminal panes — so an old client can make the new-soul probe + unachievable. When the client changed (or on a fresh checkout), run + `pnpm run build:client`; the running server picks it up on the next + browser hard refresh, with no restart. **Units and restart coupling.** With the systemd topology (`installers/systemd/freshell-rust.service` + its `managed-runtime.conf` @@ -137,8 +144,15 @@ drop-in, plus `freshell-supervisor.service`): commands below use 3001 — substitute the port the deployed unit actually sets. - Before restarting, capture the soul inventory so the post-deploy checks - can prove nothing was replaced: - `docker ps --filter name=incarnation --format '{{.ID}}\t{{.Names}}' | tee /tmp/souls-before.txt`. + can prove nothing was replaced — record container IDs AND their exact + start times (`State.StartedAt`, not `docker ps`'s `CreatedAt`, which is + the creation time and would not reveal a stop-and-restart of the same + container): + + ```bash + docker ps -q --filter name=incarnation | xargs docker inspect \ + --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | tee /tmp/souls-before.txt + ``` **What a restart does:** @@ -152,17 +166,18 @@ drop-in, plus `freshell-supervisor.service`): managed runtime live this includes SHELL panes — `shell` mode routes through souls, so shells survive the restart and reattach like any other soul-backed pane. -- **Host-side terminals end — and the client reconciles them.** Only - terminals the server hosts as its own child processes die with it: panes - created while the feature was missing (they silently spawned host-side), - panes from clients that did not negotiate `managedRuntimeV1`, and host-run - CLI panes. The process and any unsaved in-flight state are gone, but the - panes do not sit as exited terminals: on reconnect the client's reconcile - handshake replaces a lost shell with a fresh terminal automatically - (shells are stateless by design) and respawns a CLI pane with a durable - session from the CLI's own storage (`claude --resume`, `codex resume`, - opencode's session picker are the manual fallbacks when there is nothing - durable to resume). +- **Host-side terminals and sidecars end — and the client reconciles + them.** Only processes the server hosts as its own children die with it: + panes created while the feature was missing (they silently spawned + host-side), panes from clients that did not negotiate + `managedRuntimeV1`, host-run CLI panes, and any host-side fresh-agent + sidecar processes. The process and any unsaved in-flight state are gone, + but the panes do not sit as exited terminals: on reconnect the client's + reconcile handshake replaces a lost shell with a fresh terminal + automatically (shells are stateless by design) and respawns a CLI pane + with a durable session from the CLI's own storage (`claude --resume`, + `codex resume`, opencode's session picker are the manual fallbacks when + there is nothing durable to resume). **Post-deploy verification checklist:** @@ -173,10 +188,10 @@ drop-in, plus `freshell-supervisor.service`): and restart again. 3. Units: `systemctl --user is-active freshell-rust.service freshell-supervisor.service` → both `active`. -4. Existing souls: every container in the pre-restart capture is still - running with its original start time (`docker ps` with - `{{.Names}}\t{{.CreatedAt}}`-style output, or `State.StartedAt` via - `docker inspect`) — a captured container missing afterward was lost. A +4. Existing souls: re-run the pre-restart capture command and compare with + `/tmp/souls-before.txt` — every captured container ID is still running + with the SAME `State.StartedAt` (a changed `StartedAt` means the + container was stopped and restarted; a missing ID means it was lost). A container absent from the capture may appear within seconds of the restart: that is a client reconnecting and a durable terminal pane reattaching, not an old soul being replaced. @@ -184,45 +199,77 @@ drop-in, plus `freshell-supervisor.service`): the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits the separate home bind (the workspace bind already provides it), so the - checks below do not apply to that launch. Confirm a NEW - `freshell-incarnation-*` container appears (`docker ps`), that it - carries the operator-home bind and the + checks below do not apply to that launch. RECORD THE PROBE'S OWN + IDENTITY before doing anything else — other clients may create souls + concurrently, so "the newest container" is not a safe selector, and the + stop request's revision guard protects against staleness, not against + stopping the wrong soul: + + ```bash + docker ps -q --filter name=incarnation | sort > /tmp/probe-pre.txt + # ...create the probe pane now (for example via the REST tab API or the + # UI), then within seconds: + docker ps -q --filter name=incarnation | sort > /tmp/probe-post.txt + comm -13 /tmp/probe-pre.txt /tmp/probe-post.txt > /tmp/probe-new.txt + ``` + + If `/tmp/probe-new.txt` names more than one container, disambiguate by + the workspace: the probe container's mounts include the probe + workspace's directory as a bind + (`docker inspect --format '{{range .Mounts}}{{.Source}} {{end}}'`), + then overwrite `/tmp/probe-new.txt` with the single chosen ID. + Record the chosen container and its soul ID: + + ```bash + CID=$(cat /tmp/probe-new.txt) + docker inspect $CID --format '{{index .Config.Labels "com.freshell.soul-id"}}' \ + | tee /tmp/probe-soul-id.txt + ``` + + Then confirm the probe container carries the operator-home bind and the `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that its session host emitted an `operator_home_readiness` event - (`docker exec cat /run/freshell/host-lifecycle.jsonl`) - whose payload reports `"present":true` and + (`docker exec $CID cat /run/freshell/host-lifecycle.jsonl`) whose + payload reports `"present":true` and `"home_root_traversable_by_child":true` — the event is emitted with its findings even when the home is missing or unreadable, so its mere presence is NOT proof of readiness. Close the probe pane afterward. -6. Artifact freshness, per changed surface: the new probe container's image - (`docker inspect --format '{{.Image}}'`) equals the - `sha256:` ID recorded in `~/.config/freshell/managed-runtime.env`, and - every unit you restarted shows an `ExecMainStartTimestamp` - (`systemctl --user show -p ExecMainStartTimestamp`) that - postdates the artifact it was restarted to deploy. On a server-only - restart the supervisor's timestamp legitimately predates the new server - build — only compare the units that were actually bounced. + 6. Artifact freshness, per changed surface — timestamps alone don't prove + the units run the binaries you just built (an installed unit can point + at another checkout or an older executable), so verify the running + processes and mounts directly: for each restarted unit, + `readlink /proc/$(systemctl --user show -p MainPID --value)/exe` + resolves to the freshly built binary path in THIS checkout (and its + file mtime postdates your build); the new probe container's image + (`docker inspect $CID --format '{{.Image}}'`) equals the `sha256:` ID + recorded in `~/.config/freshell/managed-runtime.env`; and the probe + container's `/runtime/freshell-session-host` mount source + (`docker inspect $CID --format '{{json .Mounts}}'`) is the freshly built + `target/release/freshell-session-host` in this checkout. On a + server-only restart the supervisor's process legitimately still runs its + earlier binary — only compare the units that were actually bounced. 7. Probe cleanup: the probe soul is NOT auto-collected — the server's idle reaper explicitly skips managed terminals (their lifetime is supervisor-owned) and the supervisor has no idle reaper, so the container persists as a durable background session. To remove it - immediately, stop the soul through the fenced runtime API (find the - soul ID with `GET /api/runtime/souls`, or read the container's - `com.freshell.soul-id` docker label). Protected runtime API calls go - through OneCLI per the rollout procedure (its grant is scoped to the - server host and `/api/runtime/*` and injects `x-auth-token` — see + immediately, stop the soul recorded in step 5 — never a soul picked + fresh from `docker ps` at cleanup time — through the fenced runtime + API. Protected runtime API calls go through OneCLI per the rollout + procedure (its grant is scoped to the server host and `/api/runtime/*` + and injects `x-auth-token` — see [managed-runtime-rollout.md](managed-runtime-rollout.md)); the direct form below is the equivalent for an operator holding the checkout's token. The stop is intent-revision-guarded, and HTTP 200 alone proves nothing — the response's `outcome` must be `verified_empty` (other outcomes such as `termination_unconfirmed` or `blocked_ownership` mean - the soul is NOT confirmed stopped; re-check `GET /api/runtime/souls/$SOUL_ID` - and retry with the new revision if needed): + the soul is NOT confirmed stopped; re-check + `GET /api/runtime/souls/$SOUL_ID` and retry with the new revision if + needed): ```bash TOK=$(grep -E '^AUTH_TOKEN=' /home/dan/code/freshell/.env | cut -d= -f2-) - SOUL_ID=$(docker inspect \ - --format '{{index .Config.Labels "com.freshell.soul-id"}}') + CID=$(cat /tmp/probe-new.txt) + SOUL_ID=$(cat /tmp/probe-soul-id.txt) REV=$(curl -s -H "x-auth-token: $TOK" \ http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') From 76a1c362db6531a21e28b31f4b629b304772a5c8 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 09:52:05 -0700 Subject: [PATCH 08/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=206=20?= =?UTF-8?q?=E2=80=94=20build/restart=20skew=20sequences=20(quick=20vs=20st?= =?UTF-8?q?rict);=20terminalId-tied=20probe=20identity=20via=20soul=20reco?= =?UTF-8?q?rds;=20content-hash=20freshness=20checks=20scoped=20to=20change?= =?UTF-8?q?d=20surfaces;=20token=20path=20derived=20from=20unit=20WorkingD?= =?UTF-8?q?irectory?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 182 ++++++++++-------- 1 file changed, 99 insertions(+), 83 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 540d57fee..fd3481d44 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -96,6 +96,20 @@ cargo build --release -p freshell-server -p freshell-supervisor \ -p freshell-session-host --features freshell-server/managed-runtime-v1 ``` +Sequence the build and restart deliberately: the build rewrites +`target/release/freshell-session-host` in place, and the running supervisor +bind-mounts that path into every NEW soul — so a soul created between the +build and the restart runs the new session host against the old supervisor +and server. Two sequences: + +- Quick (compatible changes): build, then restart immediately, accepting + the short skew window. +- Strict (changes that must never skew): `systemctl --user stop + freshell-supervisor.service` (the server stops with it via the `Requires=` + coupling), build, then `systemctl --user start + freshell-supervisor.service` — no soul can be created during the build. + This is a full outage for the duration of the build. + - The server's soul controller is strictly opt-in (`default = []` in `crates/freshell-server/Cargo.toml`). Setting `FRESHELL_MANAGED_RUNTIME_V1=1` in the unit enables nothing by itself — the env var only arms a binary that @@ -195,93 +209,95 @@ drop-in, plus `freshell-supervisor.service`): container absent from the capture may appear within seconds of the restart: that is a client reconnecting and a durable terminal pane reattaching, not an old soul being replaced. -5. New-soul probe: create a terminal pane in a real workspace that is NOT - the operator home itself (a subdirectory of it is fine) — when the - workspace equals the operator home, the supervisor deliberately omits - the separate home bind (the workspace bind already provides it), so the - checks below do not apply to that launch. RECORD THE PROBE'S OWN - IDENTITY before doing anything else — other clients may create souls - concurrently, so "the newest container" is not a safe selector, and the - stop request's revision guard protects against staleness, not against - stopping the wrong soul: - - ```bash - docker ps -q --filter name=incarnation | sort > /tmp/probe-pre.txt - # ...create the probe pane now (for example via the REST tab API or the - # UI), then within seconds: - docker ps -q --filter name=incarnation | sort > /tmp/probe-post.txt - comm -13 /tmp/probe-pre.txt /tmp/probe-post.txt > /tmp/probe-new.txt - ``` - - If `/tmp/probe-new.txt` names more than one container, disambiguate by - the workspace: the probe container's mounts include the probe - workspace's directory as a bind - (`docker inspect --format '{{range .Mounts}}{{.Source}} {{end}}'`), - then overwrite `/tmp/probe-new.txt` with the single chosen ID. - Record the chosen container and its soul ID: - - ```bash - CID=$(cat /tmp/probe-new.txt) - docker inspect $CID --format '{{index .Config.Labels "com.freshell.soul-id"}}' \ - | tee /tmp/probe-soul-id.txt - ``` - - Then confirm the probe container carries the operator-home bind and the - `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that - its session host emitted an `operator_home_readiness` event - (`docker exec $CID cat /run/freshell/host-lifecycle.jsonl`) whose - payload reports `"present":true` and - `"home_root_traversable_by_child":true` — the event is emitted with its - findings even when the home is missing or unreadable, so its mere - presence is NOT proof of readiness. Close the probe pane afterward. + 5. New-soul probe: create a terminal pane in a real workspace that is NOT + the operator home itself (a subdirectory of it is fine) — when the + workspace equals the operator home, the supervisor deliberately omits + the separate home bind (the workspace bind already provides it), so the + checks below do not apply to that launch. TIE THE PROBE TO ITS OWN + IDENTITY: create the pane through the REST tab API and keep the + response's `terminalId` — every soul record carries its `terminalId`, + so the probe's soul is selected by an exact match, immune to + concurrent soul creation in the same workspace ("the newest container" + is never a safe selector, and the stop request's revision guard + protects against staleness, not against stopping the wrong soul): + + ```bash + CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) + TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) + # create the probe pane (example); record terminalId from the response: + curl -s -X POST -H "x-auth-token: $TOK" -H "Content-Type: application/json" \ + -d '{"mode":"shell","cwd":"","name":"home-mount probe"}' \ + http://127.0.0.1:3001/api/tabs | tee /tmp/probe-tab.json + TID=$(python3 -c 'import json;print(json.load(open("/tmp/probe-tab.json"))["data"]["terminalId"])') + SOUL_ID=$(curl -s -H "x-auth-token: $TOK" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys +souls=json.load(sys.stdin)["souls"] +print(next(s["soulId"] for s in souls if s.get("terminalId")=="'$TID'"))' \ + | tee /tmp/probe-soul-id.txt) + CID=$(docker ps -q --filter label=com.freshell.soul-id=$SOUL_ID) + ``` + + If the unit's environment (not the checkout's `.env`) is what supplies + the token on your installation, read `TOK` from there instead. + + Then confirm the probe container carries the operator-home bind and the + `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that + its session host emitted an `operator_home_readiness` event + (`docker exec $CID cat /run/freshell/host-lifecycle.jsonl`) whose + payload reports `"present":true` and + `"home_root_traversable_by_child":true` — the event is emitted with its + findings even when the home is missing or unreadable, so its mere + presence is NOT proof of readiness. Close the probe pane afterward. 6. Artifact freshness, per changed surface — timestamps alone don't prove - the units run the binaries you just built (an installed unit can point - at another checkout or an older executable), so verify the running - processes and mounts directly: for each restarted unit, - `readlink /proc/$(systemctl --user show -p MainPID --value)/exe` - resolves to the freshly built binary path in THIS checkout (and its - file mtime postdates your build); the new probe container's image - (`docker inspect $CID --format '{{.Image}}'`) equals the `sha256:` ID - recorded in `~/.config/freshell/managed-runtime.env`; and the probe - container's `/runtime/freshell-session-host` mount source - (`docker inspect $CID --format '{{json .Mounts}}'`) is the freshly built - `target/release/freshell-session-host` in this checkout. On a - server-only restart the supervisor's process legitimately still runs its - earlier binary — only compare the units that were actually bounced. -7. Probe cleanup: the probe soul is NOT auto-collected — the server's - idle reaper explicitly skips managed terminals (their lifetime is - supervisor-owned) and the supervisor has no idle reaper, so the - container persists as a durable background session. To remove it - immediately, stop the soul recorded in step 5 — never a soul picked - fresh from `docker ps` at cleanup time — through the fenced runtime - API. Protected runtime API calls go through OneCLI per the rollout - procedure (its grant is scoped to the server host and `/api/runtime/*` - and injects `x-auth-token` — see - [managed-runtime-rollout.md](managed-runtime-rollout.md)); the direct - form below is the equivalent for an operator holding the checkout's - token. The stop is intent-revision-guarded, and HTTP 200 alone proves - nothing — the response's `outcome` must be `verified_empty` (other - outcomes such as `termination_unconfirmed` or `blocked_ownership` mean - the soul is NOT confirmed stopped; re-check - `GET /api/runtime/souls/$SOUL_ID` and retry with the new revision if - needed): - - ```bash - TOK=$(grep -E '^AUTH_TOKEN=' /home/dan/code/freshell/.env | cut -d= -f2-) - CID=$(cat /tmp/probe-new.txt) - SOUL_ID=$(cat /tmp/probe-soul-id.txt) - REV=$(curl -s -H "x-auth-token: $TOK" \ - http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ - | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') - curl -s -X POST -H "x-auth-token: $TOK" \ - -H "Content-Type: application/json" \ - -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$REV}" \ - http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID/stop \ - | python3 -c 'import json,sys; + the units run the binaries you just built (an installed unit can point + at another checkout or an older executable, and a cached Cargo build + does not rewrite unchanged files). Verify content and mounts directly, + scoped to what this deploy actually changed. After a binary redeploy: + for each restarted unit, `readlink /proc/$(systemctl --user show + -p MainPID --value)/exe` resolves into THIS checkout, and + `sha256sum $(readlink /proc//exe)` equals `sha256sum` of the + freshly built artifact in this checkout. After an image redeploy: the + new probe container's image (`docker inspect $CID --format + '{{.Image}}'`) equals the `sha256:` ID recorded in + `~/.config/freshell/managed-runtime.env`. After a session-host + redeploy: the probe container's `/runtime/freshell-session-host` + mount source (`docker inspect $CID --format '{{json .Mounts}}'`) is + this checkout's `target/release/freshell-session-host`. On a + server-only restart the supervisor's process legitimately still runs + its earlier binary — only compare the units that were actually bounced. + 7. Probe cleanup: the probe soul is NOT auto-collected — the server's + idle reaper explicitly skips managed terminals (their lifetime is + supervisor-owned) and the supervisor has no idle reaper, so the + container persists as a durable background session. To remove it + immediately, stop the soul recorded in step 5 — never a soul picked + fresh from `docker ps` at cleanup time — through the fenced runtime + API. Protected runtime API calls go through OneCLI per the rollout + procedure (its grant is scoped to the server host and `/api/runtime/*` + and injects `x-auth-token` — see + [managed-runtime-rollout.md](managed-runtime-rollout.md)); the direct + form below is the equivalent for an operator holding the checkout's + token. The stop is intent-revision-guarded, and HTTP 200 alone proves + nothing — the response's `outcome` must be `verified_empty` (other + outcomes such as `termination_unconfirmed` or `blocked_ownership` mean + the soul is NOT confirmed stopped; re-check + `GET /api/runtime/souls/$SOUL_ID` and retry with the new revision if + needed): + + ```bash + TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) + SOUL_ID=$(cat /tmp/probe-soul-id.txt) + REV=$(curl -s -H "x-auth-token: $TOK" \ + http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') + curl -s -X POST -H "x-auth-token: $TOK" \ + -H "Content-Type: application/json" \ + -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$REV}" \ + http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID/stop \ + | python3 -c 'import json,sys; o=json.load(sys.stdin) assert o.get("outcome")=="verified_empty", o print("stopped:", o["outcome"])' - ``` + ``` Do NOT `docker stop` the container directly — the supervisor treats an externally stopped container as an unexpected exit and runs its From 6b621ff07fc0a507f8e1de27ab65ada66fda3c6e Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 10:01:55 -0700 Subject: [PATCH 09/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=207=20?= =?UTF-8?q?=E2=80=94=20strict=20sequence=20starts=20both=20units=20(Requir?= =?UTF-8?q?es=20is=20one-way);=20session-host=20content=20check=20via=20in?= =?UTF-8?q?-container=20hash;=20client=20negotiation=20applies=20to=20brow?= =?UTF-8?q?ser=20panes=20not=20the=20REST=20probe?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 30 ++++++++++++------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index fd3481d44..80e2452cb 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -106,9 +106,12 @@ and server. Two sequences: the short skew window. - Strict (changes that must never skew): `systemctl --user stop freshell-supervisor.service` (the server stops with it via the `Requires=` - coupling), build, then `systemctl --user start - freshell-supervisor.service` — no soul can be created during the build. - This is a full outage for the duration of the build. + coupling), build, then `systemctl --user start freshell-supervisor.service + freshell-rust.service` — start BOTH units: `Requires=` pulls the + supervisor in when the server starts, but starting the supervisor alone + does not start its dependent server, so starting only the supervisor + would leave the web server down. No soul can be created during the + build. This is a full outage for the duration of the build. - The server's soul controller is strictly opt-in (`default = []` in `crates/freshell-server/Cargo.toml`). Setting `FRESHELL_MANAGED_RUNTIME_V1=1` @@ -134,11 +137,13 @@ and server. Two sequences: souls use the new image; running souls keep theirs until their next incarnation. - The WEB CLIENT is a fifth deploy surface: the server serves `dist/client` - from disk, and only a client that negotiates `managedRuntimeV1` can create - managed terminal panes — so an old client can make the new-soul probe - unachievable. When the client changed (or on a fresh checkout), run - `pnpm run build:client`; the running server picks it up on the next - browser hard refresh, with no restart. + from disk, and only a browser client that negotiates `managedRuntimeV1` + gets managed terminal panes for panes it creates — so a stale client + serves its users host-side panes. (The REST probe below is NOT affected: + `POST /api/tabs` picks managed routing from the server's controller, with + no WebSocket negotiation involved.) When the client changed (or on a + fresh checkout), run `pnpm run build:client`; the running server picks + it up on the next browser hard refresh, with no restart. **Units and restart coupling.** With the systemd topology (`installers/systemd/freshell-rust.service` + its `managed-runtime.conf` @@ -259,10 +264,15 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")=="'$TID'"))' \ freshly built artifact in this checkout. After an image redeploy: the new probe container's image (`docker inspect $CID --format '{{.Image}}'`) equals the `sha256:` ID recorded in - `~/.config/freshell/managed-runtime.env`. After a session-host + `~/.config/freshell/managed-runtime.env`. After a session-host redeploy: the probe container's `/runtime/freshell-session-host` mount source (`docker inspect $CID --format '{{json .Mounts}}'`) is - this checkout's `target/release/freshell-session-host`. On a + this checkout's `target/release/freshell-session-host`, AND the + mounted bytes are the built candidate — + `docker exec $CID sha256sum /runtime/freshell-session-host` equals + `sha256sum $CHECKOUT/target/release/freshell-session-host` (the mount + is a bind of the build output, so a soul created after the build runs + the new bytes; a soul created before it keeps the old inode). On a server-only restart the supervisor's process legitimately still runs its earlier binary — only compare the units that were actually bounced. 7. Probe cleanup: the probe soul is NOT auto-collected — the server's From 6c190435a3be140785e048e6bf07e599fda220c9 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 10:13:33 -0700 Subject: [PATCH 10/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=208=20?= =?UTF-8?q?=E2=80=94=20private=20per-run=20probe=20dir=20with=20terminalId?= =?UTF-8?q?=20re-verification=20before=20stop;=20readiness=20field=20asser?= =?UTF-8?q?tions;=20execution=20outside=20the=20server=20process=20tree;?= =?UTF-8?q?=20unit-file=20install=20+=20daemon-reload;=20token=20via=20060?= =?UTF-8?q?0=20header=20file=20with=20OneCLI=20as=20the=20preferred=20scop?= =?UTF-8?q?ed=20path?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 128 +++++++++++------- 1 file changed, 78 insertions(+), 50 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 80e2452cb..e470d2e3a 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -100,7 +100,10 @@ Sequence the build and restart deliberately: the build rewrites `target/release/freshell-session-host` in place, and the running supervisor bind-mounts that path into every NEW soul — so a soul created between the build and the restart runs the new session host against the old supervisor -and server. Two sequences: +and server. Run the whole procedure from a shell OUTSIDE the server's +process tree (for example an SSH session to the host) — a host-side +Freshell terminal pane is itself killed by the restart, which would abort +the procedure mid-way and leave the service down. Two sequences: - Quick (compatible changes): build, then restart immediately, accepting the short skew window. @@ -158,6 +161,11 @@ drop-in, plus `freshell-supervisor.service`): server-only changes. - Restarting the production server always requires the user's explicit approval (`APPROVED`), including the coupled bounce above. +- When the checked-in unit files or drop-ins under `installers/systemd/` + changed, DEPLOY THEM TOO: copy them into `~/.config/systemd/user/` + (preserving the `.service.d/` drop-in layout) and run + `systemctl --user daemon-reload` before restarting — a restart alone + runs the previously installed configuration. - The deployed unit's port wins: the checked-in installer unit defaults to `PORT=3002`, while this host's installed unit serves `PORT=3001`. The commands below use 3001 — substitute the port the deployed unit actually @@ -201,10 +209,13 @@ drop-in, plus `freshell-supervisor.service`): **Post-deploy verification checklist:** 1. Health: `curl -fsS http://127.0.0.1:3001/api/health` → `"ready": true`. -2. Feature present: with the auth token, - `GET /api/runtime/readiness` answers `200`. A `404` means the running - binary was built WITHOUT the feature — rebuild with the command above - and restart again. +2. Feature present AND unblocked: with the auth token, + `GET /api/runtime/readiness` answers `200` — a `404` means the running + binary was built WITHOUT the feature; rebuild with the command above + and restart again — AND the response's `initialScanState` is + `"complete"` with an empty `blockedSubsystems` list: the handler + answers `200` even when startup recovery of existing souls is blocked, + so the status code alone is not readiness. 3. Units: `systemctl --user is-active freshell-rust.service freshell-supervisor.service` → both `active`. 4. Existing souls: re-run the pre-restart capture command and compare with @@ -214,36 +225,46 @@ drop-in, plus `freshell-supervisor.service`): container absent from the capture may appear within seconds of the restart: that is a client reconnecting and a durable terminal pane reattaching, not an old soul being replaced. - 5. New-soul probe: create a terminal pane in a real workspace that is NOT - the operator home itself (a subdirectory of it is fine) — when the - workspace equals the operator home, the supervisor deliberately omits - the separate home bind (the workspace bind already provides it), so the - checks below do not apply to that launch. TIE THE PROBE TO ITS OWN - IDENTITY: create the pane through the REST tab API and keep the - response's `terminalId` — every soul record carries its `terminalId`, - so the probe's soul is selected by an exact match, immune to - concurrent soul creation in the same workspace ("the newest container" - is never a safe selector, and the stop request's revision guard - protects against staleness, not against stopping the wrong soul): - - ```bash - CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) - TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) - # create the probe pane (example); record terminalId from the response: - curl -s -X POST -H "x-auth-token: $TOK" -H "Content-Type: application/json" \ - -d '{"mode":"shell","cwd":"","name":"home-mount probe"}' \ - http://127.0.0.1:3001/api/tabs | tee /tmp/probe-tab.json - TID=$(python3 -c 'import json;print(json.load(open("/tmp/probe-tab.json"))["data"]["terminalId"])') - SOUL_ID=$(curl -s -H "x-auth-token: $TOK" http://127.0.0.1:3001/api/runtime/souls \ - | python3 -c 'import json,sys +5. New-soul probe: create a terminal pane in a real workspace that is NOT + the operator home itself (a subdirectory of it is fine) — when the + workspace equals the operator home, the supervisor deliberately omits + the separate home bind (the workspace bind already provides it), so the + checks below do not apply to that launch. TIE THE PROBE TO ITS OWN + IDENTITY: create the pane through the REST tab API and keep the + response's `terminalId` — every soul record carries its `terminalId`, + so the probe's soul is selected by an exact match, immune to + concurrent soul creation in the same workspace ("the newest container" + is never a safe selector, and the stop request's revision guard + protects against staleness, not against stopping the wrong soul). Use a + private per-run directory — two operators (or two windows) running the + probe concurrently must never share probe files, or one run's cleanup + can stop the other run's soul: + + ```bash + D=$(mktemp -d) # private per-run directory + CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) + TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) + umask 077 + printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" # header file, not argv + # create the probe pane (example); record terminalId from the response: + curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ + -d '{"mode":"shell","cwd":"","name":"home-mount probe"}' \ + http://127.0.0.1:3001/api/tabs | tee "$D/probe-tab.json" + TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") + SOUL_ID=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] -print(next(s["soulId"] for s in souls if s.get("terminalId")=="'$TID'"))' \ - | tee /tmp/probe-soul-id.txt) - CID=$(docker ps -q --filter label=com.freshell.soul-id=$SOUL_ID) - ``` - - If the unit's environment (not the checkout's `.env`) is what supplies - the token on your installation, read `TOK` from there instead. +print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$TID" \ + | tee "$D/probe-soul-id.txt") + CID=$(docker ps -q --filter label=com.freshell.soul-id=$SOUL_ID) + ``` + + If the unit's environment (not the checkout's `.env`) is what supplies + the token on your installation, read `TOK` from there instead. The + header-file form keeps the token out of process argument lists + (visible via `/proc/*/cmdline`); it is still the full-scope token — the + OneCLI route below is the preferred, scope-limited form for anything + but a one-off operator probe. Then confirm the probe container carries the operator-home bind and the `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that @@ -280,33 +301,40 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")=="'$TID'"))' \ supervisor-owned) and the supervisor has no idle reaper, so the container persists as a durable background session. To remove it immediately, stop the soul recorded in step 5 — never a soul picked - fresh from `docker ps` at cleanup time — through the fenced runtime - API. Protected runtime API calls go through OneCLI per the rollout + fresh from `docker ps` at cleanup time — and RE-VERIFY the identity + immediately before stopping: `GET /api/runtime/souls/$SOUL_ID` must + still report the probe's `terminalId` (the one saved in `$D`); if it + does not match, stop — you are about to stop someone else's session. + Protected runtime API calls go through OneCLI per the rollout procedure (its grant is scoped to the server host and `/api/runtime/*` and injects `x-auth-token` — see - [managed-runtime-rollout.md](managed-runtime-rollout.md)); the direct - form below is the equivalent for an operator holding the checkout's - token. The stop is intent-revision-guarded, and HTTP 200 alone proves - nothing — the response's `outcome` must be `verified_empty` (other - outcomes such as `termination_unconfirmed` or `blocked_ownership` mean - the soul is NOT confirmed stopped; re-check - `GET /api/runtime/souls/$SOUL_ID` and retry with the new revision if - needed): + [managed-runtime-rollout.md](managed-runtime-rollout.md)); that is the + preferred path for anything but a one-off operator probe, because the + raw checkout token authorizes the broader API. The stop is + intent-revision-guarded, and HTTP 200 alone proves nothing — the + response's `outcome` must be `verified_empty` (other outcomes such as + `termination_unconfirmed` or `blocked_ownership` mean the soul is NOT + confirmed stopped; re-check `GET /api/runtime/souls/$SOUL_ID` and + retry with the new revision if needed): ```bash - TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) - SOUL_ID=$(cat /tmp/probe-soul-id.txt) - REV=$(curl -s -H "x-auth-token: $TOK" \ - http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ + SOUL_ID=$(cat "$D/probe-soul-id.txt") + TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") + curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ + | python3 -c 'import json,sys +s=json.load(sys.stdin)["soul"] +assert s.get("terminalId")==sys.argv[1], ("SOUL/PROBE MISMATCH", s) +print("revision:", s["intentRevision"])' "$TID" + REV=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') - curl -s -X POST -H "x-auth-token: $TOK" \ - -H "Content-Type: application/json" \ + curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$REV}" \ http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID/stop \ | python3 -c 'import json,sys; o=json.load(sys.stdin) assert o.get("outcome")=="verified_empty", o print("stopped:", o["outcome"])' + rm -rf "$D" # contains the token header file ``` Do NOT `docker stop` the container directly — the supervisor treats an From bad5097d19459e3d9e2976e5f4a140982e0ca060 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 10:19:20 -0700 Subject: [PATCH 11/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=209=20?= =?UTF-8?q?=E2=80=94=20set=20-euo=20pipefail=20so=20the=20identity=20asser?= =?UTF-8?q?t=20actually=20halts=20the=20stop;=20unit-file=20deploy=20prese?= =?UTF-8?q?rves=20site-specific=20settings=20(no=20blind=20copy);=20captur?= =?UTF-8?q?e=20pipeline=20pipefail=20+=20non-empty=20baseline;=20private?= =?UTF-8?q?=20dir=20kept=20on=20failed=20stop?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 35 ++++++++++++++----- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index e470d2e3a..8727997a4 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -162,10 +162,15 @@ drop-in, plus `freshell-supervisor.service`): - Restarting the production server always requires the user's explicit approval (`APPROVED`), including the coupled bounce above. - When the checked-in unit files or drop-ins under `installers/systemd/` - changed, DEPLOY THEM TOO: copy them into `~/.config/systemd/user/` - (preserving the `.service.d/` drop-in layout) and run - `systemctl --user daemon-reload` before restarting — a restart alone - runs the previously installed configuration. + changed, DEPLOY THEM TOO — but never blind-copy over the installed + units: they can carry site-specific settings (this host's installed + unit sets `PORT=3001` where the checked-in unit says `3002`, plus its + own `EnvironmentFile` lines). Diff the checked-in file against the + installed one, port over only the intended changes, verify the effective + unit (`systemctl --user cat freshell-rust.service + freshell-supervisor.service`), and run `systemctl --user daemon-reload` + before restarting — a restart alone runs the previously installed + configuration. - The deployed unit's port wins: the checked-in installer unit defaults to `PORT=3002`, while this host's installed unit serves `PORT=3001`. The commands below use 3001 — substitute the port the deployed unit actually @@ -177,10 +182,16 @@ drop-in, plus `freshell-supervisor.service`): container): ```bash - docker ps -q --filter name=incarnation | xargs docker inspect \ + set -o pipefail + docker ps -q --filter name=incarnation | xargs -r docker inspect \ --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | tee /tmp/souls-before.txt + test -s /tmp/souls-before.txt || echo "CAPTURE FAILED — do not proceed" ``` + `pipefail` plus the non-empty check matter: without them a Docker + failure could leave an empty baseline that the later comparison + "passes" while proving nothing. + **What a restart does:** - **Existing souls keep running.** Soul containers are owned by the Docker @@ -241,6 +252,7 @@ drop-in, plus `freshell-supervisor.service`): can stop the other run's soul: ```bash + set -euo pipefail # a failed step must halt, not continue half-named D=$(mktemp -d) # private per-run directory CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) @@ -318,25 +330,32 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T retry with the new revision if needed): ```bash + set -euo pipefail # a failed check must halt BEFORE the stop request SOUL_ID=$(cat "$D/probe-soul-id.txt") TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") + # Identity re-check FIRST; with set -e its failure aborts the block + # before the revision fetch and the stop request are ever reached: curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ | python3 -c 'import json,sys s=json.load(sys.stdin)["soul"] assert s.get("terminalId")==sys.argv[1], ("SOUL/PROBE MISMATCH", s) -print("revision:", s["intentRevision"])' "$TID" +print("identity ok:", s.get("terminalId"))' "$TID" REV=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$REV}" \ http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID/stop \ - | python3 -c 'import json,sys; + | python3 -c 'import json,sys o=json.load(sys.stdin) assert o.get("outcome")=="verified_empty", o print("stopped:", o["outcome"])' - rm -rf "$D" # contains the token header file + rm -rf "$D" # contains the token header file; runs only on success ``` + If the block aborts before `verified_empty`, `$D` is deliberately + left in place — the saved soul ID and header are what the recheck-and- + retry needs; delete `$D` manually once the probe soul is resolved. + Do NOT `docker stop` the container directly — the supervisor treats an externally stopped container as an unexpected exit and runs its automatic recovery path, which may recreate the soul or mark a session From 7d8fa6a9c21cb8e37de23f16fccdb9798cc0779d Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 10:28:29 -0700 Subject: [PATCH 12/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2010?= =?UTF-8?q?=20=E2=80=94=20post-restart=20inventory=20to=20a=20separate=20f?= =?UTF-8?q?ile=20(never=20overwrite=20the=20baseline);=20comm-based=20surv?= =?UTF-8?q?ival=20comparison;=20empty=20capture=20is=20valid,=20only=20Doc?= =?UTF-8?q?ker=20failure=20halts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 34 ++++++++++++------- 1 file changed, 21 insertions(+), 13 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 8727997a4..6576d1fc4 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -182,15 +182,13 @@ drop-in, plus `freshell-supervisor.service`): container): ```bash - set -o pipefail + set -euo pipefail # a Docker failure must halt the procedure docker ps -q --filter name=incarnation | xargs -r docker inspect \ - --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | tee /tmp/souls-before.txt - test -s /tmp/souls-before.txt || echo "CAPTURE FAILED — do not proceed" + --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort | tee /tmp/souls-before.txt ``` - `pipefail` plus the non-empty check matter: without them a Docker - failure could leave an empty baseline that the later comparison - "passes" while proving nothing. + An empty baseline is VALID (zero running souls) — `xargs -r` keeps the + empty capture successful; only an actual Docker failure halts here. **What a restart does:** @@ -229,13 +227,23 @@ drop-in, plus `freshell-supervisor.service`): so the status code alone is not readiness. 3. Units: `systemctl --user is-active freshell-rust.service freshell-supervisor.service` → both `active`. -4. Existing souls: re-run the pre-restart capture command and compare with - `/tmp/souls-before.txt` — every captured container ID is still running - with the SAME `State.StartedAt` (a changed `StartedAt` means the - container was stopped and restarted; a missing ID means it was lost). A - container absent from the capture may appear within seconds of the - restart: that is a client reconnecting and a durable terminal pane - reattaching, not an old soul being replaced. +4. Existing souls: capture the post-restart inventory to a SEPARATE file + and compare it against the preserved baseline — do NOT re-run the + pre-restart command as-is, which would overwrite `/tmp/souls-before.txt` + and compare the file against itself: + + ```bash + docker ps -q --filter name=incarnation | xargs -r docker inspect \ + --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort > /tmp/souls-after.txt + comm -23 /tmp/souls-before.txt /tmp/souls-after.txt # must be EMPTY + comm -13 /tmp/souls-before.txt /tmp/souls-after.txt # new containers (expected) + ``` + + A non-empty first `comm` is a failure: a baseline container is gone or + was restarted with a new `StartedAt`. The second `comm` lists + containers that appeared after the restart — that is a client + reconnecting and a durable terminal pane reattaching, not an old soul + being replaced. 5. New-soul probe: create a terminal pane in a real workspace that is NOT the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits From bc4c8b03e379abfc3f6a9c564b829d9b5628f45f Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 10:37:03 -0700 Subject: [PATCH 13/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2011?= =?UTF-8?q?=20=E2=80=94=20private=20per-run=20dir=20for=20survival=20snaps?= =?UTF-8?q?hots;=20explicit=20emptiness=20assertion=20on=20the=20survival?= =?UTF-8?q?=20comm;=20nested=20readiness=20field=20paths;=20honest=20attri?= =?UTF-8?q?bution=20of=20new=20containers?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 46 +++++++++++-------- 1 file changed, 27 insertions(+), 19 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 6576d1fc4..d8f2546a3 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -176,15 +176,18 @@ drop-in, plus `freshell-supervisor.service`): commands below use 3001 — substitute the port the deployed unit actually sets. - Before restarting, capture the soul inventory so the post-deploy checks - can prove nothing was replaced — record container IDs AND their exact - start times (`State.StartedAt`, not `docker ps`'s `CreatedAt`, which is - the creation time and would not reveal a stop-and-restart of the same + can prove nothing was replaced — in a private per-run directory (fixed + `/tmp` paths would let a second run overwrite the first run's baseline + before its comparison), recording container IDs AND their exact start + times (`State.StartedAt`, not `docker ps`'s `CreatedAt`, which is the + creation time and would not reveal a stop-and-restart of the same container): ```bash set -euo pipefail # a Docker failure must halt the procedure + RUN=$(mktemp -d) # private per-run dir for the survival snapshots docker ps -q --filter name=incarnation | xargs -r docker inspect \ - --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort | tee /tmp/souls-before.txt + --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort | tee "$RUN/souls-before.txt" ``` An empty baseline is VALID (zero running souls) — `xargs -r` keeps the @@ -221,29 +224,34 @@ drop-in, plus `freshell-supervisor.service`): 2. Feature present AND unblocked: with the auth token, `GET /api/runtime/readiness` answers `200` — a `404` means the running binary was built WITHOUT the feature; rebuild with the command above - and restart again — AND the response's `initialScanState` is - `"complete"` with an empty `blockedSubsystems` list: the handler - answers `200` even when startup recovery of existing souls is blocked, - so the status code alone is not readiness. + and restart again — AND the response's `readiness.initialScanState` + (the fields are nested under `readiness`) is `"complete"` with an + empty `readiness.blockedSubsystems` list: the handler answers `200` + even when startup recovery of existing souls is blocked, so the status + code alone is not readiness. 3. Units: `systemctl --user is-active freshell-rust.service freshell-supervisor.service` → both `active`. 4. Existing souls: capture the post-restart inventory to a SEPARATE file - and compare it against the preserved baseline — do NOT re-run the - pre-restart command as-is, which would overwrite `/tmp/souls-before.txt` - and compare the file against itself: + in the same private run directory and compare it against the preserved + baseline — do NOT re-run the pre-restart command as-is, which would + overwrite the baseline and compare the file against itself: ```bash docker ps -q --filter name=incarnation | xargs -r docker inspect \ - --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort > /tmp/souls-after.txt - comm -23 /tmp/souls-before.txt /tmp/souls-after.txt # must be EMPTY - comm -13 /tmp/souls-before.txt /tmp/souls-after.txt # new containers (expected) + --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort > "$RUN/souls-after.txt" + LOST=$(comm -23 "$RUN/souls-before.txt" "$RUN/souls-after.txt") + test -z "$LOST" || { echo "SOULS LOST/RESTARTED:"; echo "$LOST"; exit 1; } + comm -13 "$RUN/souls-before.txt" "$RUN/souls-after.txt" # new containers (expected) ``` - A non-empty first `comm` is a failure: a baseline container is gone or - was restarted with a new `StartedAt`. The second `comm` lists - containers that appeared after the restart — that is a client - reconnecting and a durable terminal pane reattaching, not an old soul - being replaced. + The `test -z` matters: `comm` exits successfully even when it prints + missing containers, so `set -e` alone would not enforce the empty + result. A non-empty `LOST` is a failure — a baseline container is gone + or was restarted with a new `StartedAt`. The final `comm` lists + containers that appeared after the restart; on a busy instance that is + normal (a client reconnect creating a new durable terminal, or another + client's new pane) — identify which pane or session each one belongs + to before assuming it is yours or that it is a replacement. 5. New-soul probe: create a terminal pane in a real workspace that is NOT the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits From 5f9c61fb2089d19018d31c67627819aa19c1cbd9 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 10:46:28 -0700 Subject: [PATCH 14/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2012?= =?UTF-8?q?=20=E2=80=94=20per-soul=20launchState=20verification=20(blocked?= =?UTF-8?q?=20recovery=20is=20per-soul,=20not=20in=20blockedSubsystems);?= =?UTF-8?q?=20browser-path=20check=20for=20the=20negotiated=20client=20rou?= =?UTF-8?q?te;=20token=20exception=20reconciled=20with=20OneCLI=20policy,?= =?UTF-8?q?=20trap=20scrubs=20header=20on=20every=20exit?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 54 ++++++++++++++----- 1 file changed, 41 insertions(+), 13 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index d8f2546a3..8f6d65d2c 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -221,14 +221,15 @@ drop-in, plus `freshell-supervisor.service`): **Post-deploy verification checklist:** 1. Health: `curl -fsS http://127.0.0.1:3001/api/health` → `"ready": true`. -2. Feature present AND unblocked: with the auth token, +2. Feature present AND scan completed: with the auth token, `GET /api/runtime/readiness` answers `200` — a `404` means the running binary was built WITHOUT the feature; rebuild with the command above and restart again — AND the response's `readiness.initialScanState` (the fields are nested under `readiness`) is `"complete"` with an - empty `readiness.blockedSubsystems` list: the handler answers `200` - even when startup recovery of existing souls is blocked, so the status - code alone is not readiness. + empty `readiness.blockedSubsystems` list. This proves the scan + completed without subsystem errors, NOT that every soul can reattach — + an individually blocked soul is a per-soul outcome that does not enter + `blockedSubsystems`, which is why step 4 checks each soul directly. 3. Units: `systemctl --user is-active freshell-rust.service freshell-supervisor.service` → both `active`. 4. Existing souls: capture the post-restart inventory to a SEPARATE file @@ -252,6 +253,13 @@ drop-in, plus `freshell-supervisor.service`): normal (a client reconnect creating a new durable terminal, or another client's new pane) — identify which pane or session each one belongs to before assuming it is yours or that it is a replacement. + Then check the souls INDIVIDUALLY: for each baseline soul (pull the + soul IDs from the container labels in the capture, or list + `GET /api/runtime/souls` and match by `containerId`), + `GET /api/runtime/souls/` reports `"launchState": "running"` + — a baseline soul whose launchState is anything else (blocked, lost, + stopped) cannot reattach and needs the supervisor journal and that + soul's record investigated before the deploy is called good. 5. New-soul probe: create a terminal pane in a real workspace that is NOT the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits @@ -274,6 +282,7 @@ drop-in, plus `freshell-supervisor.service`): TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) umask 077 printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" # header file, not argv + trap 'rm -f "$D/hdr"' EXIT # the token file must not survive ANY exit # create the probe pane (example); record terminalId from the response: curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d '{"mode":"shell","cwd":"","name":"home-mount probe"}' \ @@ -290,9 +299,12 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T If the unit's environment (not the checkout's `.env`) is what supplies the token on your installation, read `TOK` from there instead. The header-file form keeps the token out of process argument lists - (visible via `/proc/*/cmdline`); it is still the full-scope token — the - OneCLI route below is the preferred, scope-limited form for anything - but a one-off operator probe. + (visible via `/proc/*/cmdline`) and the `trap` scrubs it on every exit + path. This direct form is a deliberate, one-off exception for a human + operator at the host console; it uses the full-scope checkout token, + whereas every routine or agent-driven runtime API call must use the + scope-limited OneCLI grant instead (see + [managed-runtime-rollout.md](managed-runtime-rollout.md)). Then confirm the probe container carries the operator-home bind and the `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that @@ -337,8 +349,9 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T procedure (its grant is scoped to the server host and `/api/runtime/*` and injects `x-auth-token` — see [managed-runtime-rollout.md](managed-runtime-rollout.md)); that is the - preferred path for anything but a one-off operator probe, because the - raw checkout token authorizes the broader API. The stop is + policy path for routine and agent-driven calls. The direct form below + is the same one-off operator-console exception documented in step 5 — + the same `trap` scrubbing rules apply. The stop is intent-revision-guarded, and HTTP 200 alone proves nothing — the response's `outcome` must be `verified_empty` (other outcomes such as `termination_unconfirmed` or `blocked_ownership` mean the soul is NOT @@ -347,6 +360,8 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T ```bash set -euo pipefail # a failed check must halt BEFORE the stop request + D= + trap 'rm -f "$D/hdr"' EXIT # scrub the token file on every exit path SOUL_ID=$(cat "$D/probe-soul-id.txt") TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") # Identity re-check FIRST; with set -e its failure aborts the block @@ -365,18 +380,31 @@ print("identity ok:", s.get("terminalId"))' "$TID" o=json.load(sys.stdin) assert o.get("outcome")=="verified_empty", o print("stopped:", o["outcome"])' - rm -rf "$D" # contains the token header file; runs only on success + rm -rf "$D" # runs only on success; the trap already removed the header ``` - If the block aborts before `verified_empty`, `$D` is deliberately - left in place — the saved soul ID and header are what the recheck-and- - retry needs; delete `$D` manually once the probe soul is resolved. + If the block aborts before `verified_empty`, the token header is + already gone (the `trap` removed it) while the non-secret probe + records — the soul ID and the tab response — remain in `$D` for the + documented recheck-and-retry; delete `$D` manually once the probe + soul is resolved. Do NOT `docker stop` the container directly — the supervisor treats an externally stopped container as an unexpected exit and runs its automatic recovery path, which may recreate the soul or mark a session without a durable resume path as lost. Leaving the probe soul running is also fine; it is inert. + 8. Browser-path check (required when the web client was redeployed): the + REST probe in step 5 deliberately bypasses WebSocket capability + negotiation, so it can pass while a stale browser client still creates + host-side panes. Hard-refresh the web client, create a terminal pane + FROM THE UI, and confirm that pane also gets a soul: snapshot + `GET /api/runtime/souls` before and after, and a newly appeared soul + matching the moment of pane creation is the pass; no new soul (and no + host-side process instead) means the client is stale or not + negotiating `managedRuntimeV1` — rebuild/redeploy the client and + hard-refresh again. Close the test pane (and stop its soul per step 7 + if desired). ## Launch the native Windows client From cc8c58dcba0b6d19a9ca5f42b9accbc2e8f435f2 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 10:56:11 -0700 Subject: [PATCH 15/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2013?= =?UTF-8?q?=20=E2=80=94=20executable=20D=20recovery=20(live=20var=20+=20po?= =?UTF-8?q?inter=20file,=20same-shell=20contract);=20self-healing=20token?= =?UTF-8?q?=20header=20for=20the=20retry;=20browser-path=20check=20require?= =?UTF-8?q?d=20on=20every=20redeploy,=20proven=20by=20terminalCwd=20match?= =?UTF-8?q?=20not=20timing?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 44 ++++++++++++++----- 1 file changed, 34 insertions(+), 10 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 8f6d65d2c..b8d05acca 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -278,6 +278,7 @@ drop-in, plus `freshell-supervisor.service`): ```bash set -euo pipefail # a failed step must halt, not continue half-named D=$(mktemp -d) # private per-run directory + echo "$D" > "$HOME/.freshell/managed-runtime-probe-dir" # recovery pointer CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) umask 077 @@ -306,6 +307,10 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T scope-limited OneCLI grant instead (see [managed-runtime-rollout.md](managed-runtime-rollout.md)). + Run steps 5-7 in ONE shell session: the private directory `$D` and its + records persist between them. If the shell is lost, the pointer file + `~/.freshell/managed-runtime-probe-dir` names the directory. + Then confirm the probe container carries the operator-home bind and the `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that its session host emitted an `operator_home_readiness` event @@ -360,7 +365,14 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T ```bash set -euo pipefail # a failed check must halt BEFORE the stop request - D= + D=${D:-$(cat "$HOME/.freshell/managed-runtime-probe-dir")} + CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) + # Self-healing retry: the trap removed the header on the previous exit; + # recreate it (0600) only when missing: + if [ ! -f "$D/hdr" ]; then + TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) + umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" + fi trap 'rm -f "$D/hdr"' EXIT # scrub the token file on every exit path SOUL_ID=$(cat "$D/probe-soul-id.txt") TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") @@ -394,17 +406,29 @@ print("stopped:", o["outcome"])' automatic recovery path, which may recreate the soul or mark a session without a durable resume path as lost. Leaving the probe soul running is also fine; it is inert. - 8. Browser-path check (required when the web client was redeployed): the + 8. Browser-path check (required on EVERY managed-runtime redeploy — not + only after a client redeploy: any reconnected client can be stale): the REST probe in step 5 deliberately bypasses WebSocket capability negotiation, so it can pass while a stale browser client still creates - host-side panes. Hard-refresh the web client, create a terminal pane - FROM THE UI, and confirm that pane also gets a soul: snapshot - `GET /api/runtime/souls` before and after, and a newly appeared soul - matching the moment of pane creation is the pass; no new soul (and no - host-side process instead) means the client is stale or not - negotiating `managedRuntimeV1` — rebuild/redeploy the client and - hard-refresh again. Close the test pane (and stop its soul per step 7 - if desired). + host-side panes. Hard-refresh the web client, then create a terminal + pane FROM THE UI in a distinctive throwaway workspace used only by + this check (for example `mkdir -p ~/scratch/browser-path-check` and + point the pane at it). Prove that exact pane is managed by matching + its workspace, not its timing — a soul appearing "around the same + moment" proves nothing under concurrency: + + ```bash + curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys +souls=json.load(sys.stdin)["souls"] +hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1] and s.get("launchState")=="running"] +assert hits, "no managed soul for the UI pane — client is stale or not negotiating" +print("UI pane is managed:", hits[0]["soulId"])' ~/scratch/browser-path-check + ``` + + If the assertion fails, the client is stale or not negotiating + `managedRuntimeV1` — rebuild/redeploy the client and hard-refresh + again. Close the test pane (and stop its soul per step 7 if desired). ## Launch the native Windows client From 9b9e0163c6620e30520f2d04a3ee999698f4f7b3 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 11:06:29 -0700 Subject: [PATCH 16/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2014?= =?UTF-8?q?=20=E2=80=94=20browser=20check=20before=20cleanup=20(D/hdr=20st?= =?UTF-8?q?ill=20alive);=20unique=20per-run=20workspaces=20for=20both=20pr?= =?UTF-8?q?obes;=20shared=20pointer=20removed=20for=20cwd-based=20lost-she?= =?UTF-8?q?ll=20recovery;=20labels=20wording=20fixed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 110 ++++++++++-------- 1 file changed, 61 insertions(+), 49 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index b8d05acca..2839c31ba 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -253,9 +253,10 @@ drop-in, plus `freshell-supervisor.service`): normal (a client reconnect creating a new durable terminal, or another client's new pane) — identify which pane or session each one belongs to before assuming it is yours or that it is a replacement. - Then check the souls INDIVIDUALLY: for each baseline soul (pull the - soul IDs from the container labels in the capture, or list - `GET /api/runtime/souls` and match by `containerId`), + Then check the souls INDIVIDUALLY: for each baseline container in the + capture, read its soul ID with + `docker inspect --format '{{index .Config.Labels "com.freshell.soul-id"}}'` + (or list `GET /api/runtime/souls` and match by `containerId`); `GET /api/runtime/souls/` reports `"launchState": "running"` — a baseline soul whose launchState is anything else (blocked, lost, stopped) cannot reattach and needs the supervisor journal and that @@ -278,7 +279,8 @@ drop-in, plus `freshell-supervisor.service`): ```bash set -euo pipefail # a failed step must halt, not continue half-named D=$(mktemp -d) # private per-run directory - echo "$D" > "$HOME/.freshell/managed-runtime-probe-dir" # recovery pointer + mkdir -p ~/scratch + WS=$(mktemp -d ~/scratch/home-mount-probe.XXXXXX) # unique probe workspace CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) umask 077 @@ -286,7 +288,7 @@ drop-in, plus `freshell-supervisor.service`): trap 'rm -f "$D/hdr"' EXIT # the token file must not survive ANY exit # create the probe pane (example); record terminalId from the response: curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ - -d '{"mode":"shell","cwd":"","name":"home-mount probe"}' \ + -d '{"mode":"shell","cwd":"'"$WS"'","name":"home-mount probe"}' \ http://127.0.0.1:3001/api/tabs | tee "$D/probe-tab.json" TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") SOUL_ID=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ @@ -307,9 +309,15 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T scope-limited OneCLI grant instead (see [managed-runtime-rollout.md](managed-runtime-rollout.md)). - Run steps 5-7 in ONE shell session: the private directory `$D` and its - records persist between them. If the shell is lost, the pointer file - `~/.freshell/managed-runtime-probe-dir` names the directory. + Run steps 5-8 in ONE shell session: the private directory `$D`, the + probe workspace `$WS`, and their records persist between them. If the + shell is lost, do NOT guess at recovery from any shared state: the + probe soul is re-derivable only by its unique workspace — find the + soul whose `terminalCwd` equals `$WS` in `GET /api/runtime/souls` and + stop it with the same fenced request, then delete the directories by + hand. The probe workspace under `~/scratch/` is unique per run + (`mktemp -d`), which is also what makes workspace-based matching safe + from cross-run and cross-operator confusion. Then confirm the probe container carries the operator-home bind and the `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that @@ -341,15 +349,42 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T the new bytes; a soul created before it keeps the old inode). On a server-only restart the supervisor's process legitimately still runs its earlier binary — only compare the units that were actually bounced. - 7. Probe cleanup: the probe soul is NOT auto-collected — the server's - idle reaper explicitly skips managed terminals (their lifetime is - supervisor-owned) and the supervisor has no idle reaper, so the - container persists as a durable background session. To remove it - immediately, stop the soul recorded in step 5 — never a soul picked - fresh from `docker ps` at cleanup time — and RE-VERIFY the identity - immediately before stopping: `GET /api/runtime/souls/$SOUL_ID` must - still report the probe's `terminalId` (the one saved in `$D`); if it - does not match, stop — you are about to stop someone else's session. + 7. Browser-path check (required on EVERY managed-runtime redeploy — not + only after a client redeploy: any reconnected client can be stale): the + REST probe in step 5 deliberately bypasses WebSocket capability + negotiation, so it can pass while a stale browser client still creates + host-side panes. Run it BEFORE the cleanup step so the probe records + and token header still exist. Hard-refresh the web client, create a + terminal pane FROM THE UI in a fresh unique workspace minted for this + run — never a reused fixed path, which a stale soul from an earlier + redeploy could match: + + ```bash + BWS=$(mktemp -d ~/scratch/browser-path-check.XXXXXX) # unique per run + # ...hard-refresh the web client, create a terminal pane at $BWS... + curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys +souls=json.load(sys.stdin)["souls"] +hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1] and s.get("launchState")=="running"] +assert hits, "no managed soul for the UI pane — client is stale or not negotiating" +print("UI pane is managed:", hits[0]["soulId"])' "$BWS" + ``` + + If the assertion fails, the client is stale or not negotiating + `managedRuntimeV1` — rebuild/redeploy the client and hard-refresh + again. Close the test pane. + 8. Probe cleanup (LAST): probe souls are NOT auto-collected — the + server's idle reaper explicitly skips managed terminals (their + lifetime is supervisor-owned) and the supervisor has no idle reaper, + so the containers persist as durable background sessions. To remove + them, stop each through the fenced runtime API. For the REST probe + soul, use the records from step 5 — never a soul picked fresh from + `docker ps` at cleanup time — and RE-VERIFY the identity immediately + before stopping: `GET /api/runtime/souls/$SOUL_ID` must still report + the probe's `terminalId` (the one saved in `$D`); if it does not + match, stop — you are about to stop someone else's session. For the + browser-check soul, locate it by its unique workspace + (`terminalCwd == $BWS`) the same way step 7 did. Protected runtime API calls go through OneCLI per the rollout procedure (its grant is scoped to the server host and `/api/runtime/*` and injects `x-auth-token` — see @@ -365,7 +400,6 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T ```bash set -euo pipefail # a failed check must halt BEFORE the stop request - D=${D:-$(cat "$HOME/.freshell/managed-runtime-probe-dir")} CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) # Self-healing retry: the trap removed the header on the previous exit; # recreate it (0600) only when missing: @@ -398,37 +432,15 @@ print("stopped:", o["outcome"])' If the block aborts before `verified_empty`, the token header is already gone (the `trap` removed it) while the non-secret probe records — the soul ID and the tab response — remain in `$D` for the - documented recheck-and-retry; delete `$D` manually once the probe - soul is resolved. - - Do NOT `docker stop` the container directly — the supervisor treats an - externally stopped container as an unexpected exit and runs its - automatic recovery path, which may recreate the soul or mark a session - without a durable resume path as lost. Leaving the probe soul running - is also fine; it is inert. - 8. Browser-path check (required on EVERY managed-runtime redeploy — not - only after a client redeploy: any reconnected client can be stale): the - REST probe in step 5 deliberately bypasses WebSocket capability - negotiation, so it can pass while a stale browser client still creates - host-side panes. Hard-refresh the web client, then create a terminal - pane FROM THE UI in a distinctive throwaway workspace used only by - this check (for example `mkdir -p ~/scratch/browser-path-check` and - point the pane at it). Prove that exact pane is managed by matching - its workspace, not its timing — a soul appearing "around the same - moment" proves nothing under concurrency: - - ```bash - curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ - | python3 -c 'import json,sys -souls=json.load(sys.stdin)["souls"] -hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1] and s.get("launchState")=="running"] -assert hits, "no managed soul for the UI pane — client is stale or not negotiating" -print("UI pane is managed:", hits[0]["soulId"])' ~/scratch/browser-path-check - ``` - - If the assertion fails, the client is stale or not negotiating - `managedRuntimeV1` — rebuild/redeploy the client and hard-refresh - again. Close the test pane (and stop its soul per step 7 if desired). + documented recheck-and-retry; delete `$D`, `$WS`, and `$BWS` + manually once the probe souls are resolved. On success, also remove + the workspaces: `rm -rf "$D" "$WS" "$BWS"`. + + Do NOT `docker stop` a probe container directly — the supervisor + treats an externally stopped container as an unexpected exit and runs + its automatic recovery path, which may recreate the soul or mark a + session without a durable resume path as lost. Leaving a probe soul + running is also fine; it is inert. ## Launch the native Windows client From 2c99067a0b2ab7aa5faad91a002fdffbadcc2a51 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 11:15:24 -0700 Subject: [PATCH 17/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2015?= =?UTF-8?q?=20=E2=80=94=20probe=20workspaces=20chmod=200770=20(in-soul=20s?= =?UTF-8?q?hell=2065534:0=20needs=20group=20access);=20run.env=20record=20?= =?UTF-8?q?+=20pointer=20with=20explicit=20set-e/lost-shell=20recovery=20v?= =?UTF-8?q?ia=20workspace=20timestamps?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 32 ++++++++++++++----- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 2839c31ba..bdb749ab8 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -281,11 +281,16 @@ drop-in, plus `freshell-supervisor.service`): D=$(mktemp -d) # private per-run directory mkdir -p ~/scratch WS=$(mktemp -d ~/scratch/home-mount-probe.XXXXXX) # unique probe workspace + chmod 0770 "$WS" # the in-soul shell runs as 65534:0 — group-traversable + # and group-writable is required or the probe terminal + # cannot use its own workspace CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) umask 077 printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" # header file, not argv trap 'rm -f "$D/hdr"' EXIT # the token file must not survive ANY exit + printf 'D=%q\nWS=%q\nCHECKOUT=%q\n' "$D" "$WS" "$CHECKOUT" > "$D/run.env" + echo "$D" > "$HOME/.freshell/managed-runtime-probe-dir" # recovery pointer # create the probe pane (example); record terminalId from the response: curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d '{"mode":"shell","cwd":"'"$WS"'","name":"home-mount probe"}' \ @@ -310,14 +315,17 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T [managed-runtime-rollout.md](managed-runtime-rollout.md)). Run steps 5-8 in ONE shell session: the private directory `$D`, the - probe workspace `$WS`, and their records persist between them. If the - shell is lost, do NOT guess at recovery from any shared state: the - probe soul is re-derivable only by its unique workspace — find the - soul whose `terminalCwd` equals `$WS` in `GET /api/runtime/souls` and - stop it with the same fenced request, then delete the directories by - hand. The probe workspace under `~/scratch/` is unique per run - (`mktemp -d`), which is also what makes workspace-based matching safe - from cross-run and cross-operator confusion. + probe workspace `$WS`, and their records persist between them, and + `$D/run.env` plus the `~/.freshell/managed-runtime-probe-dir` pointer + let a later block reload them (see step 8's recovery lines). If the + shell is lost or a check aborts under `set -e`, the FILES survive — + only the variables die — so recovery is: open a new shell, then either + source the pointer's `run.env`, or, when concurrent probe runs make + the pointer untrustworthy, locate YOUR workspace by timestamp + (`ls -dt ~/scratch/home-mount-probe.*`) and recover purely from it: + the soul whose `terminalCwd` equals that workspace IS your probe + (workspace uniqueness is the identity proof), so apply the same + fenced stop to that soul directly and delete the directories by hand. Then confirm the probe container carries the operator-home bind and the `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that @@ -361,6 +369,8 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T ```bash BWS=$(mktemp -d ~/scratch/browser-path-check.XXXXXX) # unique per run + chmod 0770 "$BWS" # same in-soul-shell access requirement as $WS + echo "BWS=$BWS" >> "$D/run.env" # ...hard-refresh the web client, create a terminal pane at $BWS... curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys @@ -400,6 +410,12 @@ print("UI pane is managed:", hits[0]["soulId"])' "$BWS" ```bash set -euo pipefail # a failed check must halt BEFORE the stop request + # Reload this run's records if the shell died between steps (files + # survive a set -e abort; only the variables die): + if [ -z "${D:-}" ] && [ -f "$HOME/.freshell/managed-runtime-probe-dir" ]; then + . "$(cat "$HOME/.freshell/managed-runtime-probe-dir")/run.env" + fi + : "${D:?no run records — use the workspace-timestamp recovery above}" CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) # Self-healing retry: the trap removed the header on the previous exit; # recreate it (0600) only when missing: From 7e7e66267040f5509a2009227090d62f2694dee2 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 11:24:30 -0700 Subject: [PATCH 18/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2016?= =?UTF-8?q?=20=E2=80=94=20strict=20start=20waits=20for=20the=20supervisor?= =?UTF-8?q?=20control=20socket=20(Type=3Dsimple=20binds=20late);=20socket?= =?UTF-8?q?=20check=20in=20checklist;=20browser-probe=20soul=20fenced-stop?= =?UTF-8?q?ped=20before=20its=20workspace=20is=20deleted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 60 +++++++++++++++---- 1 file changed, 50 insertions(+), 10 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index bdb749ab8..689b05c0e 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -109,12 +109,29 @@ the procedure mid-way and leave the service down. Two sequences: the short skew window. - Strict (changes that must never skew): `systemctl --user stop freshell-supervisor.service` (the server stops with it via the `Requires=` - coupling), build, then `systemctl --user start freshell-supervisor.service - freshell-rust.service` — start BOTH units: `Requires=` pulls the - supervisor in when the server starts, but starting the supervisor alone - does not start its dependent server, so starting only the supervisor - would leave the web server down. No soul can be created during the - build. This is a full outage for the duration of the build. + coupling), build, then start in socket order: start BOTH units — + `Requires=` pulls the supervisor in when the server starts, but starting + the supervisor alone does not start its dependent server, so starting + only the supervisor would leave the web server down. The supervisor + unit is `Type=simple`, so systemd considers it "started" BEFORE its + control socket exists (the socket binds after startup reconciliation); + the server needs that socket. After starting the supervisor, wait for + the socket before relying on the server, or start the server in a + second command once the socket is live: + + ```bash + systemctl --user start freshell-supervisor.service + until [ -S "$HOME/.freshell/managed-runtime/control/supervisor.sock" ]; do + sleep 1 + done + systemctl --user start freshell-rust.service + ``` + + If the server unit fails its first start in the race window, that is + this socket ordering, not a bad build — wait for the socket and start + it again (repeated immediate failures can also trip the unit's start + limit). No soul can be created during the build. This is a full outage + for the duration of the build. - The server's soul controller is strictly opt-in (`default = []` in `crates/freshell-server/Cargo.toml`). Setting `FRESHELL_MANAGED_RUNTIME_V1=1` @@ -231,7 +248,11 @@ drop-in, plus `freshell-supervisor.service`): an individually blocked soul is a per-soul outcome that does not enter `blockedSubsystems`, which is why step 4 checks each soul directly. 3. Units: `systemctl --user is-active freshell-rust.service - freshell-supervisor.service` → both `active`. + freshell-supervisor.service` → both `active`, and the supervisor's + control socket + (`~/.freshell/managed-runtime/control/supervisor.sock`) exists with + mode `0600` — the supervisor unit is `Type=simple`, so "active" does + not by itself prove the socket is up. 4. Existing souls: capture the post-restart inventory to a SEPARATE file in the same private run directory and compare it against the preserved baseline — do NOT re-run the pre-restart command as-is, which would @@ -442,15 +463,34 @@ print("identity ok:", s.get("terminalId"))' "$TID" o=json.load(sys.stdin) assert o.get("outcome")=="verified_empty", o print("stopped:", o["outcome"])' - rm -rf "$D" # runs only on success; the trap already removed the header + # Stop the browser-check soul the same way BEFORE deleting its + # workspace (closing the pane only detaches; the soul keeps running): + if [ -n "${BWS:-}" ]; then + BS_ID=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys +souls=json.load(sys.stdin)["souls"] +hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1] and s.get("launchState")=="running"] +print(hits[0]["soulId"] if hits else "")' "$BWS") + if [ -n "$BS_ID" ]; then + BREV=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$BS_ID \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') + curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ + -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$BREV}" \ + http://127.0.0.1:3001/api/runtime/souls/$BS_ID/stop \ + | python3 -c 'import json,sys +o=json.load(sys.stdin) +assert o.get("outcome")=="verified_empty", o +print("stopped browser soul:", o["outcome"])' + fi + fi + rm -rf "$D" "$WS" "${BWS:-}" # runs only on success; trap removed the header ``` If the block aborts before `verified_empty`, the token header is already gone (the `trap` removed it) while the non-secret probe records — the soul ID and the tab response — remain in `$D` for the documented recheck-and-retry; delete `$D`, `$WS`, and `$BWS` - manually once the probe souls are resolved. On success, also remove - the workspaces: `rm -rf "$D" "$WS" "$BWS"`. + manually once BOTH probe souls are resolved. Do NOT `docker stop` a probe container directly — the supervisor treats an externally stopped container as an unexpected exit and runs From f8f6e76362d15fe58bf324ec0205cdb53035dc3f Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 11:30:59 -0700 Subject: [PATCH 19/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2017?= =?UTF-8?q?=20=E2=80=94=20connect-probe=20socket=20liveness=20(stale=20soc?= =?UTF-8?q?ket=20files);=20same=20race=20noted=20on=20coupled=20bounce;=20?= =?UTF-8?q?pointer=20recovery=20removed=20for=20WS-based=20identity;=20bro?= =?UTF-8?q?wser-soul=20handled=20in=20any=20launchState;=20robust=20dotenv?= =?UTF-8?q?=20token=20extraction?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 167 ++++++++++++------ 1 file changed, 116 insertions(+), 51 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 689b05c0e..be94470ce 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -121,17 +121,28 @@ the procedure mid-way and leave the service down. Two sequences: ```bash systemctl --user start freshell-supervisor.service - until [ -S "$HOME/.freshell/managed-runtime/control/supervisor.sock" ]; do + SOCK="$HOME/.freshell/managed-runtime/control/supervisor.sock" + for i in $(seq 1 60); do + # CONNECT-probe, not a file test: a stopped supervisor leaves its old + # socket FILE on disk, so [ -S ... ] passes on a dead listener: + python3 -c 'import socket,sys +s=socket.socket(socket.AF_UNIX); s.settimeout(1) +try: s.connect(sys.argv[1]) +except OSError: sys.exit(1)' "$SOCK" 2>/dev/null && break + systemctl --user is-active freshell-supervisor.service >/dev/null \ + || { echo "supervisor unit is not active"; exit 1; } sleep 1 done systemctl --user start freshell-rust.service ``` - If the server unit fails its first start in the race window, that is - this socket ordering, not a bad build — wait for the socket and start - it again (repeated immediate failures can also trip the unit's start - limit). No soul can be created during the build. This is a full outage - for the duration of the build. + The connect-probe waits for a LIVE listener (a stale file fails the + connect), fails fast if the supervisor unit dies, and is bounded. If the + server unit fails its first start in a race window, that is this socket + ordering, not a bad build — wait for the live socket and start it again + (repeated immediate failures can also trip the unit's start limit). No + soul can be created during the build. This is a full outage for the + duration of the build. - The server's soul controller is strictly opt-in (`default = []` in `crates/freshell-server/Cargo.toml`). Setting `FRESHELL_MANAGED_RUNTIME_V1=1` @@ -173,6 +184,11 @@ drop-in, plus `freshell-supervisor.service`): `freshell-supervisor.service`. Restarting the SUPERVISOR therefore also restarts the server in the same transaction — one `systemctl --user restart freshell-supervisor.service` redeploys both. + The same `Type=simple` socket race as the strict sequence applies here: + the server may start before the supervisor rebinds its control socket + and come up failed. If the server unit is not `active` after the + coupled bounce, connect-probe the socket as in the strict sequence and + `systemctl --user start freshell-rust.service` once it is live. - Restarting the server alone (`systemctl --user restart freshell-rust.service`) leaves the supervisor running. Use this for server-only changes. @@ -250,9 +266,10 @@ drop-in, plus `freshell-supervisor.service`): 3. Units: `systemctl --user is-active freshell-rust.service freshell-supervisor.service` → both `active`, and the supervisor's control socket - (`~/.freshell/managed-runtime/control/supervisor.sock`) exists with - mode `0600` — the supervisor unit is `Type=simple`, so "active" does - not by itself prove the socket is up. + (`~/.freshell/managed-runtime/control/supervisor.sock`) is LIVE — + connect-probe it as in the strict sequence, because the supervisor unit + is `Type=simple` and a stopped supervisor leaves its old socket file + on disk. 4. Existing souls: capture the post-restart inventory to a SEPARATE file in the same private run directory and compare it against the preserved baseline — do NOT re-run the pre-restart command as-is, which would @@ -306,13 +323,22 @@ drop-in, plus `freshell-supervisor.service`): # and group-writable is required or the probe terminal # cannot use its own workspace CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) - TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) + # Robust dotenv extraction (the server accepts `export AUTH_TOKEN=`, + # quoted values, and inline comments on unquoted values — a bare + # `grep AUTH_TOKEN= | cut` mangles all three): + TOK=$(python3 -c 'import re,sys +m=re.search(r"^\s*(?:export\s+)?AUTH_TOKEN\s*=\s*(.*)$", open(sys.argv[1]).read(), re.M) +v=m.group(1).strip() +if v[:1] in (chr(34),chr(39)): + v=v[1:-1] if v[-1:]==v[0] else v +else: + v=v.split(chr(35))[0].strip() +print(v)' "$CHECKOUT/.env") umask 077 printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" # header file, not argv trap 'rm -f "$D/hdr"' EXIT # the token file must not survive ANY exit - printf 'D=%q\nWS=%q\nCHECKOUT=%q\n' "$D" "$WS" "$CHECKOUT" > "$D/run.env" - echo "$D" > "$HOME/.freshell/managed-runtime-probe-dir" # recovery pointer - # create the probe pane (example); record terminalId from the response: + printf 'D=%q\nWS=%q\nCHECKOUT=%q\n' "$D" "$WS" "$CHECKOUT" > "$D/run.env" + # create the probe pane (example); record terminalId from the response: curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d '{"mode":"shell","cwd":"'"$WS"'","name":"home-mount probe"}' \ http://127.0.0.1:3001/api/tabs | tee "$D/probe-tab.json" @@ -336,17 +362,20 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T [managed-runtime-rollout.md](managed-runtime-rollout.md)). Run steps 5-8 in ONE shell session: the private directory `$D`, the - probe workspace `$WS`, and their records persist between them, and - `$D/run.env` plus the `~/.freshell/managed-runtime-probe-dir` pointer - let a later block reload them (see step 8's recovery lines). If the + probe workspace `$WS`, and their records persist between them. If the shell is lost or a check aborts under `set -e`, the FILES survive — - only the variables die — so recovery is: open a new shell, then either - source the pointer's `run.env`, or, when concurrent probe runs make - the pointer untrustworthy, locate YOUR workspace by timestamp - (`ls -dt ~/scratch/home-mount-probe.*`) and recover purely from it: - the soul whose `terminalCwd` equals that workspace IS your probe - (workspace uniqueness is the identity proof), so apply the same - fenced stop to that soul directly and delete the directories by hand. + only the variables die. There is deliberately NO shared pointer file to + reload: a pointer can be overwritten by a concurrent probe run whose + records are self-consistent, and loading it would stop the wrong soul. + Recovery is by YOUR unique workspace only — locate it by timestamp + (`ls -dt ~/scratch/home-mount-probe.*`), pick the one that is yours + (you know when you ran the probe; inspect the directory if unsure), + and recover purely from it: the soul whose `terminalCwd` equals that + workspace IS your probe (workspace uniqueness is the identity proof), + so apply the same fenced stop to that soul directly, then delete the + directories by hand. If you cannot identify your workspace with + certainty, do not stop anything — leave the probe soul running (it is + inert) rather than risk stopping another run's session. Then confirm the probe container carries the operator-home bind and the `FRESHELL_OPERATOR_HOME_MOUNT=1` marker env (`docker inspect`), and that @@ -431,31 +460,62 @@ print("UI pane is managed:", hits[0]["soulId"])' "$BWS" ```bash set -euo pipefail # a failed check must halt BEFORE the stop request - # Reload this run's records if the shell died between steps (files - # survive a set -e abort; only the variables die): - if [ -z "${D:-}" ] && [ -f "$HOME/.freshell/managed-runtime-probe-dir" ]; then - . "$(cat "$HOME/.freshell/managed-runtime-probe-dir")/run.env" - fi - : "${D:?no run records — use the workspace-timestamp recovery above}" CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) - # Self-healing retry: the trap removed the header on the previous exit; - # recreate it (0600) only when missing: + if [ -z "${D:-}" ]; then + # Lost-shell recovery — deliberately NOT pointer-based (a concurrent + # run's records are self-consistent and would stop the wrong soul). + # $WS is the identity proof; export it from YOUR workspace listing: + if [ -z "${WS:-}" ]; then + ls -dt ~/scratch/home-mount-probe.* 2>/dev/null || true + echo "export WS=, then re-run" + exit 1 + fi + D=$(mktemp -d) # fresh records dir for the recovery run + fi + # Self-healing token header: the trap removed it on the previous exit; + # recreate it (0600) only when missing, with the same robust dotenv + # extraction as step 5: if [ ! -f "$D/hdr" ]; then - TOK=$(grep -E '^AUTH_TOKEN=' "$CHECKOUT/.env" | cut -d= -f2-) + TOK=$(python3 -c 'import re,sys +m=re.search(r"^\s*(?:export\s+)?AUTH_TOKEN\s*=\s*(.*)$", open(sys.argv[1]).read(), re.M) +v=m.group(1).strip() +if v[:1] in (chr(34),chr(39)): + v=v[1:-1] if v[-1:]==v[0] else v +else: + v=v.split(chr(35))[0].strip() +print(v)' "$CHECKOUT/.env") umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" fi trap 'rm -f "$D/hdr"' EXIT # scrub the token file on every exit path - SOUL_ID=$(cat "$D/probe-soul-id.txt") - TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") - # Identity re-check FIRST; with set -e its failure aborts the block - # before the revision fetch and the stop request are ever reached: - curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ - | python3 -c 'import json,sys + if [ -z "${SOUL_ID:-}" ]; then + # Recovery mode: derive the probe soul from the unique workspace — + # no independent terminalId check applies here; WS uniqueness IS + # the identity proof. (Requires WS; a known $D alone is NOT enough — + # its records are only trustworthy alongside this shell's own step 5.) + if [ -z "${WS:-}" ]; then + ls -dt ~/scratch/home-mount-probe.* 2>/dev/null || true + echo "export WS=, then re-run" + exit 1 + fi + PROBE=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys +souls=json.load(sys.stdin)["souls"] +hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] +assert hits, "no soul for this workspace — nothing to stop" +print(hits[0]["soulId"], hits[0]["intentRevision"], hits[0]["terminalId"])' "$WS") + read -r SOUL_ID REV TID <<< "$PROBE" + else + TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") + # Identity re-check FIRST; with set -e its failure aborts the block + # before the revision fetch and the stop request are ever reached: + curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ + | python3 -c 'import json,sys s=json.load(sys.stdin)["soul"] assert s.get("terminalId")==sys.argv[1], ("SOUL/PROBE MISMATCH", s) print("identity ok:", s.get("terminalId"))' "$TID" - REV=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ - | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') + REV=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') + fi curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$REV}" \ http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID/stop \ @@ -464,23 +524,28 @@ o=json.load(sys.stdin) assert o.get("outcome")=="verified_empty", o print("stopped:", o["outcome"])' # Stop the browser-check soul the same way BEFORE deleting its - # workspace (closing the pane only detaches; the soul keeps running): + # workspace (closing the pane only detaches; the soul keeps running). + # Look it up by workspace in ANY launchState — a blocked or exited + # probe soul still must not be orphaned against a deleted workspace: if [ -n "${BWS:-}" ]; then - BS_ID=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + BS=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] -hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1] and s.get("launchState")=="running"] -print(hits[0]["soulId"] if hits else "")' "$BWS") - if [ -n "$BS_ID" ]; then - BREV=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$BS_ID \ - | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') - curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ - -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$BREV}" \ - http://127.0.0.1:3001/api/runtime/souls/$BS_ID/stop \ - | python3 -c 'import json,sys +hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] +print((hits[0]["soulId"], hits[0]["intentRevision"], hits[0]["launchState"]) if hits else "")' "$BWS") + if [ -n "$BS" ]; then + BS_ID=$(cut -d" " -f1 <<<"$BS") + BS_REV=$(cut -d" " -f2 <<<"$BS") + BS_STATE=$(cut -d" " -f3 <<<"$BS") + if [ "$BS_STATE" != "stopped" ]; then + curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ + -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$BS_REV}" \ + http://127.0.0.1:3001/api/runtime/souls/$BS_ID/stop \ + | python3 -c 'import json,sys o=json.load(sys.stdin) assert o.get("outcome")=="verified_empty", o print("stopped browser soul:", o["outcome"])' + fi fi fi rm -rf "$D" "$WS" "${BWS:-}" # runs only on success; trap removed the header From d14ed2d044dab29ffb63be2e06428ff33e8d7a35 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 11:41:16 -0700 Subject: [PATCH 20/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2018?= =?UTF-8?q?=20=E2=80=94=20loop=20exhaustion=20re-verified=20before=20serve?= =?UTF-8?q?r=20start;=20token=20read=20from=20the=20running=20server's=20/?= =?UTF-8?q?proc=20environ=20(no=20dotenv=20re-parsing);=20space-separated?= =?UTF-8?q?=20lookup=20output;=20newest-incarnation=20selection;=20BWS=20i?= =?UTF-8?q?ncluded=20in=20recovery=20prompt?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 87 ++++++++++--------- 1 file changed, 48 insertions(+), 39 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index be94470ce..66c235c3b 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -125,20 +125,29 @@ the procedure mid-way and leave the service down. Two sequences: for i in $(seq 1 60); do # CONNECT-probe, not a file test: a stopped supervisor leaves its old # socket FILE on disk, so [ -S ... ] passes on a dead listener: - python3 -c 'import socket,sys + if python3 -c 'import socket,sys s=socket.socket(socket.AF_UNIX); s.settimeout(1) try: s.connect(sys.argv[1]) -except OSError: sys.exit(1)' "$SOCK" 2>/dev/null && break +except OSError: sys.exit(1)' "$SOCK" 2>/dev/null; then + break + fi systemctl --user is-active freshell-supervisor.service >/dev/null \ || { echo "supervisor unit is not active"; exit 1; } sleep 1 done + # The loop alone proves nothing if it exhausted — verify before starting: + python3 -c 'import socket,sys +s=socket.socket(socket.AF_UNIX); s.settimeout(1) +try: s.connect(sys.argv[1]) +except OSError: sys.exit(1)' "$SOCK" 2>/dev/null \ + || { echo "supervisor socket never became live"; exit 1; } systemctl --user start freshell-rust.service ``` The connect-probe waits for a LIVE listener (a stale file fails the - connect), fails fast if the supervisor unit dies, and is bounded. If the - server unit fails its first start in a race window, that is this socket + connect), fails fast if the supervisor unit dies, and the final probe + refuses to start the server if the socket never came up. If the server + unit fails its first start in a race window, that is this socket ordering, not a bad build — wait for the live socket and start it again (repeated immediate failures can also trip the unit's start limit). No soul can be created during the build. This is a full outage for the @@ -323,17 +332,12 @@ drop-in, plus `freshell-supervisor.service`): # and group-writable is required or the probe terminal # cannot use its own workspace CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) - # Robust dotenv extraction (the server accepts `export AUTH_TOKEN=`, - # quoted values, and inline comments on unquoted values — a bare - # `grep AUTH_TOKEN= | cut` mangles all three): - TOK=$(python3 -c 'import re,sys -m=re.search(r"^\s*(?:export\s+)?AUTH_TOKEN\s*=\s*(.*)$", open(sys.argv[1]).read(), re.M) -v=m.group(1).strip() -if v[:1] in (chr(34),chr(39)): - v=v[1:-1] if v[-1:]==v[0] else v -else: - v=v.split(chr(35))[0].strip() -print(v)' "$CHECKOUT/.env") + # The token the RUNNING server actually validates against — read from + # its own environment, not by re-parsing .env (the server's dotenv + # accepts export prefixes, quotes, and comment forms that naive + # grep/cut extraction mangles): + SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) + TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p') umask 077 printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" # header file, not argv trap 'rm -f "$D/hdr"' EXIT # the token file must not survive ANY exit @@ -352,13 +356,14 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T ``` If the unit's environment (not the checkout's `.env`) is what supplies - the token on your installation, read `TOK` from there instead. The - header-file form keeps the token out of process argument lists - (visible via `/proc/*/cmdline`) and the `trap` scrubs it on every exit - path. This direct form is a deliberate, one-off exception for a human - operator at the host console; it uses the full-scope checkout token, - whereas every routine or agent-driven runtime API call must use the - scope-limited OneCLI grant instead (see + the token on your installation, the `/proc` read above already picks + that up — it reads the server's effective environment, whatever its + source. The header-file form keeps the token out of process argument + lists (visible via `/proc/*/cmdline`) and the `trap` scrubs it on + every exit path. This direct form is a deliberate, one-off exception + for a human operator at the host console; it uses the full-scope + checkout token, whereas every routine or agent-driven runtime API call + must use the scope-limited OneCLI grant instead (see [managed-runtime-rollout.md](managed-runtime-rollout.md)). Run steps 5-8 in ONE shell session: the private directory `$D`, the @@ -464,26 +469,24 @@ print("UI pane is managed:", hits[0]["soulId"])' "$BWS" if [ -z "${D:-}" ]; then # Lost-shell recovery — deliberately NOT pointer-based (a concurrent # run's records are self-consistent and would stop the wrong soul). - # $WS is the identity proof; export it from YOUR workspace listing: + # $WS is the identity proof; export it from YOUR workspace listing. + # If your run included the step-7 browser check, export BWS too — + # otherwise its soul is reported below but deliberately left + # running rather than guessed at: if [ -z "${WS:-}" ]; then ls -dt ~/scratch/home-mount-probe.* 2>/dev/null || true - echo "export WS=, then re-run" + ls -dt ~/scratch/browser-path-check.* 2>/dev/null || true + echo "export WS=, and BWS= if applicable, then re-run" exit 1 fi D=$(mktemp -d) # fresh records dir for the recovery run fi # Self-healing token header: the trap removed it on the previous exit; - # recreate it (0600) only when missing, with the same robust dotenv - # extraction as step 5: + # recreate it (0600) only when missing, reading the token from the + # RUNNING server's environment exactly as step 5 does: if [ ! -f "$D/hdr" ]; then - TOK=$(python3 -c 'import re,sys -m=re.search(r"^\s*(?:export\s+)?AUTH_TOKEN\s*=\s*(.*)$", open(sys.argv[1]).read(), re.M) -v=m.group(1).strip() -if v[:1] in (chr(34),chr(39)): - v=v[1:-1] if v[-1:]==v[0] else v -else: - v=v.split(chr(35))[0].strip() -print(v)' "$CHECKOUT/.env") + SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) + TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p') umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" fi trap 'rm -f "$D/hdr"' EXIT # scrub the token file on every exit path @@ -502,7 +505,8 @@ print(v)' "$CHECKOUT/.env") souls=json.load(sys.stdin)["souls"] hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] assert hits, "no soul for this workspace — nothing to stop" -print(hits[0]["soulId"], hits[0]["intentRevision"], hits[0]["terminalId"])' "$WS") +s=hits[-1] # newest incarnation (inventory is oldest-first) +print(s["soulId"], s["intentRevision"], s["terminalId"])' "$WS") read -r SOUL_ID REV TID <<< "$PROBE" else TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") @@ -528,15 +532,20 @@ print("stopped:", o["outcome"])' # Look it up by workspace in ANY launchState — a blocked or exited # probe soul still must not be orphaned against a deleted workspace: if [ -n "${BWS:-}" ]; then + # Select the NEWEST record for this workspace (the inventory lists + # incarnations oldest first, so after a recovery the newest entry is + # the live one; an older stopped incarnation may also match): BS=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -print((hits[0]["soulId"], hits[0]["intentRevision"], hits[0]["launchState"]) if hits else "")' "$BWS") +if hits: + s=hits[-1] + print(s["soulId"], s["intentRevision"], s["launchState"]) +else: + print("")' "$BWS") if [ -n "$BS" ]; then - BS_ID=$(cut -d" " -f1 <<<"$BS") - BS_REV=$(cut -d" " -f2 <<<"$BS") - BS_STATE=$(cut -d" " -f3 <<<"$BS") + read -r BS_ID BS_REV BS_STATE <<< "$BS" if [ "$BS_STATE" != "stopped" ]; then curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$BS_REV}" \ From b5bc7900c38392d644d83240f9e711613432ad9e Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 11:53:44 -0700 Subject: [PATCH 21/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2019?= =?UTF-8?q?=20=E2=80=94=20900s=20socket=20bound=20(reconciliation=20is=201?= =?UTF-8?q?80s/soul=20at=204-way);=20per-soul=20check=20asserts=20launchSt?= =?UTF-8?q?ate=20AND=20recoveryState;=20token:=20environ=20primary=20+=20d?= =?UTF-8?q?otenvy-parity=20.env=20fallback=20+=20non-empty=20assert?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 66 +++++++++++++++---- 1 file changed, 55 insertions(+), 11 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 66c235c3b..475c868fe 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -122,7 +122,11 @@ the procedure mid-way and leave the service down. Two sequences: ```bash systemctl --user start freshell-supervisor.service SOCK="$HOME/.freshell/managed-runtime/control/supervisor.sock" - for i in $(seq 1 60); do + # The supervisor binds its socket only AFTER startup reconciliation, + # which can take minutes (default 180s per soul, four concurrent + # recoveries — ten souls can legitimately take ~8 minutes). Bound the + # wait generously (900s), not tightly: + for i in $(seq 1 900); do # CONNECT-probe, not a file test: a stopped supervisor leaves its old # socket FILE on disk, so [ -S ... ] passes on a dead listener: if python3 -c 'import socket,sys @@ -304,10 +308,12 @@ drop-in, plus `freshell-supervisor.service`): capture, read its soul ID with `docker inspect --format '{{index .Config.Labels "com.freshell.soul-id"}}'` (or list `GET /api/runtime/souls` and match by `containerId`); - `GET /api/runtime/souls/` reports `"launchState": "running"` - — a baseline soul whose launchState is anything else (blocked, lost, - stopped) cannot reattach and needs the supervisor journal and that - soul's record investigated before the deploy is called good. + `GET /api/runtime/souls/` reports BOTH `"launchState": + "running"` AND a `recoveryState` that is neither `"blocked"` nor + `"lost"` — those are `recoveryState` values, so `launchState` can + stay `"running"` while the soul cannot reattach; a blocked or lost + soul needs the supervisor journal and its record investigated before + the deploy is called good. 5. New-soul probe: create a terminal pane in a real workspace that is NOT the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits @@ -332,12 +338,32 @@ drop-in, plus `freshell-supervisor.service`): # and group-writable is required or the probe terminal # cannot use its own workspace CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) - # The token the RUNNING server actually validates against — read from - # its own environment, not by re-parsing .env (the server's dotenv - # accepts export prefixes, quotes, and comment forms that naive - # grep/cut extraction mangles): + # Token source: the server's INITIAL environment first (exact for + # unit-supplied tokens), then a fallback parse of the checkout's .env + # following the server's dotenv rules for the common forms (fully + # quoted values, quoted-then-comment, unquoted with a + # whitespace-preceded `#` comment; `v#x` keeps the `#` literally). + # NOTE: /proc//environ does NOT contain variables dotenvy loaded + # at runtime — that is exactly why the .env fallback exists. SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p') + if [ -z "$TOK" ]; then + TOK=$(python3 - "$CHECKOUT/.env" <<'PY' +import re,sys +last=None +for line in open(sys.argv[1]): + m=re.match(r'^\s*(?:export\s+)?AUTH_TOKEN\s*=\s*(.*)$', line.rstrip("\n")) + if m: last=m.group(1) +if last is None: sys.exit(0) +if last[:1] in ('"',"'"): + end=last.find(last[0],1) + print(last[1:end] if end>0 else last[1:]) +else: + print(re.split(r'\s#', last.strip(), 1)[0].strip()) +PY +) + fi + test -n "$TOK" || { echo "AUTH_TOKEN not found — extract it manually"; exit 1; } umask 077 printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" # header file, not argv trap 'rm -f "$D/hdr"' EXIT # the token file must not survive ANY exit @@ -482,11 +508,29 @@ print("UI pane is managed:", hits[0]["soulId"])' "$BWS" D=$(mktemp -d) # fresh records dir for the recovery run fi # Self-healing token header: the trap removed it on the previous exit; - # recreate it (0600) only when missing, reading the token from the - # RUNNING server's environment exactly as step 5 does: + # recreate it (0600) only when missing, extracting the token exactly + # as step 5 does (initial environment first, .env dotenv fallback, + # hard non-empty assert): if [ ! -f "$D/hdr" ]; then SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p') + if [ -z "$TOK" ]; then + TOK=$(python3 - "$CHECKOUT/.env" <<'PY' +import re,sys +last=None +for line in open(sys.argv[1]): + m=re.match(r'^\s*(?:export\s+)?AUTH_TOKEN\s*=\s*(.*)$', line.rstrip("\n")) + if m: last=m.group(1) +if last is None: sys.exit(0) +if last[:1] in ('"',"'"): + end=last.find(last[0],1) + print(last[1:end] if end>0 else last[1:]) +else: + print(re.split(r'\s#', last.strip(), 1)[0].strip()) +PY +) + fi + test -n "$TOK" || { echo "AUTH_TOKEN not found — extract it manually"; exit 1; } umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" fi trap 'rm -f "$D/hdr"' EXIT # scrub the token file on every exit path From ff2139da47bb665d14b83a1584d45fcc1936b6e8 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 12:02:28 -0700 Subject: [PATCH 22/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2020?= =?UTF-8?q?=20=E2=80=94=20inventory-sized=20socket=20bound=20from=20the=20?= =?UTF-8?q?pre-restart=20capture;=20~/scratch=20group-traversable=20for=20?= =?UTF-8?q?in-soul=20path=20traversal?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 475c868fe..6d3d2b3cd 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -123,10 +123,12 @@ the procedure mid-way and leave the service down. Two sequences: systemctl --user start freshell-supervisor.service SOCK="$HOME/.freshell/managed-runtime/control/supervisor.sock" # The supervisor binds its socket only AFTER startup reconciliation, - # which can take minutes (default 180s per soul, four concurrent - # recoveries — ten souls can legitimately take ~8 minutes). Bound the - # wait generously (900s), not tightly: - for i in $(seq 1 900); do + # which runs up to 180s per soul at four-way concurrency — size the + # bound from the actual inventory (the pre-restart capture) instead of + # a fixed guess: 45s/soul effective + a 5-minute margin: + N=$(wc -l < "$RUN/souls-before.txt" 2>/dev/null || echo 0) + BOUND=$(( N * 45 + 300 )) + for i in $(seq 1 "$BOUND"); do # CONNECT-probe, not a file test: a stopped supervisor leaves its old # socket FILE on disk, so [ -S ... ] passes on a dead listener: if python3 -c 'import socket,sys @@ -333,10 +335,13 @@ drop-in, plus `freshell-supervisor.service`): set -euo pipefail # a failed step must halt, not continue half-named D=$(mktemp -d) # private per-run directory mkdir -p ~/scratch + chmod 0770 ~/scratch # the in-soul provider (65534:0) must traverse + # EVERY path component down to the workspace — + # a 0700 ~/scratch fails the probe for reasons + # unrelated to the redeployed runtime WS=$(mktemp -d ~/scratch/home-mount-probe.XXXXXX) # unique probe workspace - chmod 0770 "$WS" # the in-soul shell runs as 65534:0 — group-traversable - # and group-writable is required or the probe terminal - # cannot use its own workspace + chmod 0770 "$WS" # group-traversable + group-writable, or the probe + # terminal cannot use its own workspace CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) # Token source: the server's INITIAL environment first (exact for # unit-supplied tokens), then a fallback parse of the checkout's .env From 19d6a634ce9638e6d615ead4e64189b954ad7f28 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 12:12:04 -0700 Subject: [PATCH 23/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2021?= =?UTF-8?q?=20=E2=80=94=20preliminary=20baseline=20capture=20(containers?= =?UTF-8?q?=20+=20desired-running=20soul=20IDs)=20before=20the=20sequences?= =?UTF-8?q?;=20socket=20bound=20sized=20from=20desired-running=20inventory?= =?UTF-8?q?=20with=20container=20fallback;=20per-soul=20sweep=20over=20the?= =?UTF-8?q?=20desired-running=20list?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 137 +++++++++++++----- 1 file changed, 98 insertions(+), 39 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 6d3d2b3cd..367ab206f 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -96,6 +96,61 @@ cargo build --release -p freshell-server -p freshell-supervisor \ -p freshell-session-host --features freshell-server/managed-runtime-v1 ``` +**Preliminary: capture the baseline.** Before restarting anything, +capture two snapshots into a private per-run directory (fixed `/tmp` +paths would let a second run overwrite the first run's baseline before +its comparison): the RUNNING containers (their IDs and exact start +times) and the DESIRED-RUNNING soul IDs — the supervisor reconciles +every desired-running soul, including ones with no running container, so +that second list is what sizes the strict-sequence wait and the +post-deploy soul sweep: + +```bash +set -euo pipefail +RUN=$(mktemp -d) # private per-run dir for the survival snapshots +docker ps -q --filter name=incarnation | xargs -r docker inspect \ + --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort | tee "$RUN/souls-before.txt" +CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) +SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) +TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p') +if [ -z "$TOK" ]; then + TOK=$(python3 - "$CHECKOUT/.env" <<'PY' +import re,sys +last=None +for line in open(sys.argv[1]): + m=re.match(r'^\s*(?:export\s+)?AUTH_TOKEN\s*=\s*(.*)$', line.rstrip("\n")) + if m: last=m.group(1) +if last is None: sys.exit(0) +if last[:1] in ('"',"'"): + end=last.find(last[0],1) + print(last[1:end] if end>0 else last[1:]) +else: + print(re.split(r'\s#', last.strip(), 1)[0].strip()) +PY +) +fi +test -n "$TOK" || { echo "AUTH_TOKEN not found — extract it manually"; exit 1; } +umask 077 +printf 'x-auth-token: %s\n' "$TOK" > "$RUN/hdr" +trap 'rm -f "$RUN/hdr"' EXIT +if curl -s -o /dev/null -w '%{http_code}' -H @"$RUN/hdr" \ + http://127.0.0.1:3001/api/runtime/readiness | grep -q 200; then + curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys +souls=json.load(sys.stdin)["souls"] +for s in souls: + if s.get("desiredState")=="running": + print(s["soulId"])' > "$RUN/desired-running.txt" +else + echo "runtime API unavailable (featureless or stopped server) — container count only" >&2 + : > "$RUN/desired-running.txt" +fi +``` + +An empty container snapshot is VALID (zero running souls); an empty +`desired-running.txt` from the API-less fallback only narrows the +strict-sequence estimate (see below). + Sequence the build and restart deliberately: the build rewrites `target/release/freshell-session-host` in place, and the running supervisor bind-mounts that path into every NEW soul — so a soul created between the @@ -123,10 +178,13 @@ the procedure mid-way and leave the service down. Two sequences: systemctl --user start freshell-supervisor.service SOCK="$HOME/.freshell/managed-runtime/control/supervisor.sock" # The supervisor binds its socket only AFTER startup reconciliation, - # which runs up to 180s per soul at four-way concurrency — size the - # bound from the actual inventory (the pre-restart capture) instead of - # a fixed guess: 45s/soul effective + a 5-minute margin: - N=$(wc -l < "$RUN/souls-before.txt" 2>/dev/null || echo 0) + # which runs up to 180s per soul at four-way concurrency. Size the + # bound from the DESIRED-RUNNING inventory (the reconciliation work + # list — including souls with no running container), falling back to + # the container count when the API was unavailable: 45s/soul + # effective + a 5-minute margin: + N=$(wc -l < "$RUN/desired-running.txt" 2>/dev/null || echo 0) + [ "$N" -gt 0 ] || N=$(wc -l < "$RUN/souls-before.txt" 2>/dev/null || echo 0) BOUND=$(( N * 45 + 300 )) for i in $(seq 1 "$BOUND"); do # CONNECT-probe, not a file test: a stopped supervisor leaves its old @@ -223,23 +281,9 @@ drop-in, plus `freshell-supervisor.service`): `PORT=3002`, while this host's installed unit serves `PORT=3001`. The commands below use 3001 — substitute the port the deployed unit actually sets. -- Before restarting, capture the soul inventory so the post-deploy checks - can prove nothing was replaced — in a private per-run directory (fixed - `/tmp` paths would let a second run overwrite the first run's baseline - before its comparison), recording container IDs AND their exact start - times (`State.StartedAt`, not `docker ps`'s `CreatedAt`, which is the - creation time and would not reveal a stop-and-restart of the same - container): - - ```bash - set -euo pipefail # a Docker failure must halt the procedure - RUN=$(mktemp -d) # private per-run dir for the survival snapshots - docker ps -q --filter name=incarnation | xargs -r docker inspect \ - --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort | tee "$RUN/souls-before.txt" - ``` - - An empty baseline is VALID (zero running souls) — `xargs -r` keeps the - empty capture successful; only an actual Docker failure halts here. +- The soul-inventory baselines were already captured in the preliminary + step above (`$RUN/souls-before.txt` and `$RUN/desired-running.txt`); the + post-deploy checks in the checklist compare against those. **What a restart does:** @@ -298,24 +342,39 @@ drop-in, plus `freshell-supervisor.service`): comm -13 "$RUN/souls-before.txt" "$RUN/souls-after.txt" # new containers (expected) ``` - The `test -z` matters: `comm` exits successfully even when it prints - missing containers, so `set -e` alone would not enforce the empty - result. A non-empty `LOST` is a failure — a baseline container is gone - or was restarted with a new `StartedAt`. The final `comm` lists - containers that appeared after the restart; on a busy instance that is - normal (a client reconnect creating a new durable terminal, or another - client's new pane) — identify which pane or session each one belongs - to before assuming it is yours or that it is a replacement. - Then check the souls INDIVIDUALLY: for each baseline container in the - capture, read its soul ID with - `docker inspect --format '{{index .Config.Labels "com.freshell.soul-id"}}'` - (or list `GET /api/runtime/souls` and match by `containerId`); - `GET /api/runtime/souls/` reports BOTH `"launchState": - "running"` AND a `recoveryState` that is neither `"blocked"` nor - `"lost"` — those are `recoveryState` values, so `launchState` can - stay `"running"` while the soul cannot reattach; a blocked or lost - soul needs the supervisor journal and its record investigated before - the deploy is called good. + The `test -z` matters: `comm` exits successfully even when it prints + missing containers, so `set -e` alone would not enforce the empty + result. A non-empty `LOST` is a failure — a baseline container is gone + or was restarted with a new `StartedAt`. The final `comm` lists + containers that appeared after the restart; on a busy instance that is + normal (a client reconnect creating a new durable terminal, or another + client's new pane) — identify which pane or session each one belongs + to before assuming it is yours or that it is a replacement. + Then check the souls INDIVIDUALLY, over the DESIRED-RUNNING list from + the preliminary capture — not just the containers (a desired-running + soul with no running container at capture time still had to + reconcile, and it is exactly the kind the container comparison never + sees): + + ```bash + while read -r SID; do + [ -n "$SID" ] || continue + curl -s -H @"$RUN/hdr" "http://127.0.0.1:3001/api/runtime/souls/$SID" \ + | python3 -c 'import json,sys +s=json.load(sys.stdin)["soul"] +ls=s.get("launchState"); rs=s.get("recoveryState") +assert ls=="running" and rs not in ("blocked","lost"), (sys.argv[1], ls, rs) +print("ok:", sys.argv[1])' "$SID" + done < "$RUN/desired-running.txt" + ``` + + A blocked or lost `recoveryState` can hide behind a `"running"` + `launchState` — that is why both fields are asserted. Any failure here + needs the supervisor journal and that soul's record investigated + before the deploy is called good. If the preliminary capture fell back + to an empty `desired-running.txt` (featureless or stopped server), the + per-soul sweep is unavailable — re-run it after the deploy against the + freshly served soul list. 5. New-soul probe: create a terminal pane in a real workspace that is NOT the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits From 2eadd90fd81dd1c9e6b536dfcedf7977d40db43e Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 12:26:17 -0700 Subject: [PATCH 24/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2022?= =?UTF-8?q?=20=E2=80=94=20liveness-checked=20unbounded=20socket=20wait=20(?= =?UTF-8?q?config=20knobs=20make=20fixed=20bounds=20wrong);=20single=20com?= =?UTF-8?q?bined=20EXIT=20trap=20scrubs=20both=20token=20headers;=20probe?= =?UTF-8?q?=20tree=20under=20a=20fresh=20per-run=20root=20(never=20chmod?= =?UTF-8?q?=20pre-existing=20dirs);=20post-deploy=20refresh=20of=20the=20d?= =?UTF-8?q?esired-running=20sweep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 100 ++++++++++-------- 1 file changed, 55 insertions(+), 45 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 367ab206f..898fd3754 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -177,16 +177,13 @@ the procedure mid-way and leave the service down. Two sequences: ```bash systemctl --user start freshell-supervisor.service SOCK="$HOME/.freshell/managed-runtime/control/supervisor.sock" - # The supervisor binds its socket only AFTER startup reconciliation, - # which runs up to 180s per soul at four-way concurrency. Size the - # bound from the DESIRED-RUNNING inventory (the reconciliation work - # list — including souls with no running container), falling back to - # the container count when the API was unavailable: 45s/soul - # effective + a 5-minute margin: - N=$(wc -l < "$RUN/desired-running.txt" 2>/dev/null || echo 0) - [ "$N" -gt 0 ] || N=$(wc -l < "$RUN/souls-before.txt" 2>/dev/null || echo 0) - BOUND=$(( N * 45 + 300 )) - for i in $(seq 1 "$BOUND"); do + # The supervisor binds its socket only AFTER startup reconciliation, and + # reconciliation sizing is CONFIGURABLE (FRESHELL_RUNTIME_STARTUP_CONCURRENCY, + # per-soul timeout up to 900s, plus pending loss cleanup) — so no fixed + # formula can bound it reliably. Wait on supervisor-unit LIVENESS as the + # failure detector instead: a healthy-but-slow supervisor is not a + # failure, a dead unit is: + while true; do # CONNECT-probe, not a file test: a stopped supervisor leaves its old # socket FILE on disk, so [ -S ... ] passes on a dead listener: if python3 -c 'import socket,sys @@ -197,25 +194,19 @@ except OSError: sys.exit(1)' "$SOCK" 2>/dev/null; then fi systemctl --user is-active freshell-supervisor.service >/dev/null \ || { echo "supervisor unit is not active"; exit 1; } - sleep 1 + sleep 5 done - # The loop alone proves nothing if it exhausted — verify before starting: - python3 -c 'import socket,sys -s=socket.socket(socket.AF_UNIX); s.settimeout(1) -try: s.connect(sys.argv[1]) -except OSError: sys.exit(1)' "$SOCK" 2>/dev/null \ - || { echo "supervisor socket never became live"; exit 1; } systemctl --user start freshell-rust.service ``` The connect-probe waits for a LIVE listener (a stale file fails the - connect), fails fast if the supervisor unit dies, and the final probe - refuses to start the server if the socket never came up. If the server - unit fails its first start in a race window, that is this socket - ordering, not a bad build — wait for the live socket and start it again - (repeated immediate failures can also trip the unit's start limit). No - soul can be created during the build. This is a full outage for the - duration of the build. + connect) and the unit-liveness check is the real failure detector — the + loop ends when the socket appears or the supervisor dies, whatever the + reconciliation duration. If the server unit fails its first start in a + race window, that is this socket ordering, not a bad build — wait for + the live socket and start it again (repeated immediate failures can + also trip the unit's start limit). No soul can be created during the + build. This is a full outage for the duration of the build. - The server's soul controller is strictly opt-in (`default = []` in `crates/freshell-server/Cargo.toml`). Setting `FRESHELL_MANAGED_RUNTIME_V1=1` @@ -357,6 +348,18 @@ drop-in, plus `freshell-supervisor.service`): sees): ```bash + # If the preliminary capture fell back to an empty list (featureless + # or stopped server), refresh it NOW from the freshly deployed API — + # an empty sweep would otherwise pass vacuously: + if [ ! -s "$RUN/desired-running.txt" ]; then + curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys +souls=json.load(sys.stdin)["souls"] +for s in souls: + if s.get("desiredState")=="running": + print(s["soulId"])' > "$RUN/desired-running.txt" + fi + echo "sweeping $(wc -l < "$RUN/desired-running.txt") desired-running soul(s)" while read -r SID; do [ -n "$SID" ] || continue curl -s -H @"$RUN/hdr" "http://127.0.0.1:3001/api/runtime/souls/$SID" \ @@ -371,10 +374,9 @@ print("ok:", sys.argv[1])' "$SID" A blocked or lost `recoveryState` can hide behind a `"running"` `launchState` — that is why both fields are asserted. Any failure here needs the supervisor journal and that soul's record investigated - before the deploy is called good. If the preliminary capture fell back - to an empty `desired-running.txt` (featureless or stopped server), the - per-soul sweep is unavailable — re-run it after the deploy against the - freshly served soul list. + before the deploy is called good. A zero-soul sweep after a + SUCCESSFUL refresh is a legitimate empty instance — but the printed + count makes that visible instead of vacuous. 5. New-soul probe: create a terminal pane in a real workspace that is NOT the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits @@ -393,14 +395,16 @@ print("ok:", sys.argv[1])' "$SID" ```bash set -euo pipefail # a failed step must halt, not continue half-named D=$(mktemp -d) # private per-run directory - mkdir -p ~/scratch - chmod 0770 ~/scratch # the in-soul provider (65534:0) must traverse - # EVERY path component down to the workspace — - # a 0700 ~/scratch fails the probe for reasons - # unrelated to the redeployed runtime - WS=$(mktemp -d ~/scratch/home-mount-probe.XXXXXX) # unique probe workspace - chmod 0770 "$WS" # group-traversable + group-writable, or the probe - # terminal cannot use its own workspace + # Probe workspaces live under a FRESH per-run root directly under the + # home root (group-traversable after the host prep) — never chmod a + # pre-existing directory like ~/scratch, which may hold unrelated + # data; we only ever change permissions of directories this run + # created: + PROBE_ROOT=$(mktemp -d ~/freshell-probe.XXXXXX) + chmod 0770 "$PROBE_ROOT" # the in-soul provider (65534:0) must + # traverse and write what we create + WS=$(mktemp -d "$PROBE_ROOT/ws.XXXXXX") # unique probe workspace + chmod 0770 "$WS" CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) # Token source: the server's INITIAL environment first (exact for # unit-supplied tokens), then a fallback parse of the checkout's .env @@ -430,7 +434,9 @@ PY test -n "$TOK" || { echo "AUTH_TOKEN not found — extract it manually"; exit 1; } umask 077 printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" # header file, not argv - trap 'rm -f "$D/hdr"' EXIT # the token file must not survive ANY exit + # ONE EXIT trap per shell: this REPLACES the preliminary capture's + # trap, so scrub BOTH token headers here or $RUN/hdr would leak: + trap 'rm -f "$RUN/hdr" "$D/hdr"' EXIT printf 'D=%q\nWS=%q\nCHECKOUT=%q\n' "$D" "$WS" "$CHECKOUT" > "$D/run.env" # create the probe pane (example); record terminalId from the response: curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ @@ -463,7 +469,7 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T reload: a pointer can be overwritten by a concurrent probe run whose records are self-consistent, and loading it would stop the wrong soul. Recovery is by YOUR unique workspace only — locate it by timestamp - (`ls -dt ~/scratch/home-mount-probe.*`), pick the one that is yours + (`ls -dt ~/freshell-probe.*/ws.*`), pick the one that is yours (you know when you ran the probe; inspect the directory if unsure), and recover purely from it: the soul whose `terminalCwd` equals that workspace IS your probe (workspace uniqueness is the identity proof), @@ -513,7 +519,9 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T redeploy could match: ```bash - BWS=$(mktemp -d ~/scratch/browser-path-check.XXXXXX) # unique per run + : "${PROBE_ROOT:=$(mktemp -d ~/freshell-probe.XXXXXX)}" + chmod 0770 "$PROBE_ROOT" # only ever chmods a directory this run made + BWS=$(mktemp -d "$PROBE_ROOT/bws.XXXXXX") # unique per run chmod 0770 "$BWS" # same in-soul-shell access requirement as $WS echo "BWS=$BWS" >> "$D/run.env" # ...hard-refresh the web client, create a terminal pane at $BWS... @@ -564,8 +572,8 @@ print("UI pane is managed:", hits[0]["soulId"])' "$BWS" # otherwise its soul is reported below but deliberately left # running rather than guessed at: if [ -z "${WS:-}" ]; then - ls -dt ~/scratch/home-mount-probe.* 2>/dev/null || true - ls -dt ~/scratch/browser-path-check.* 2>/dev/null || true + ls -dt ~/freshell-probe.*/ws.* 2>/dev/null || true + ls -dt ~/freshell-probe.*/bws.* 2>/dev/null || true echo "export WS=, and BWS= if applicable, then re-run" exit 1 fi @@ -597,15 +605,16 @@ PY test -n "$TOK" || { echo "AUTH_TOKEN not found — extract it manually"; exit 1; } umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" fi - trap 'rm -f "$D/hdr"' EXIT # scrub the token file on every exit path + trap 'rm -f "$RUN/hdr" "$D/hdr"' EXIT # BOTH token headers, every exit path if [ -z "${SOUL_ID:-}" ]; then # Recovery mode: derive the probe soul from the unique workspace — # no independent terminalId check applies here; WS uniqueness IS # the identity proof. (Requires WS; a known $D alone is NOT enough — # its records are only trustworthy alongside this shell's own step 5.) if [ -z "${WS:-}" ]; then - ls -dt ~/scratch/home-mount-probe.* 2>/dev/null || true - echo "export WS=, then re-run" + ls -dt ~/freshell-probe.*/ws.* 2>/dev/null || true + ls -dt ~/freshell-probe.*/bws.* 2>/dev/null || true + echo "export WS=, and BWS= if applicable, then re-run" exit 1 fi PROBE=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ @@ -665,7 +674,8 @@ print("stopped browser soul:", o["outcome"])' fi fi fi - rm -rf "$D" "$WS" "${BWS:-}" # runs only on success; trap removed the header + rm -rf "$D" "${PROBE_ROOT:-}" "$WS" "${BWS:-}" # success only; both + # token headers were already scrubbed by the trap ``` If the block aborts before `verified_empty`, the token header is From a771022eb523bb62ca10eee3395d41eeaf44d66d Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 12:35:22 -0700 Subject: [PATCH 25/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2023?= =?UTF-8?q?=20=E2=80=94=20set-u-safe=20traps=20(no=20header=20leak=20in=20?= =?UTF-8?q?recovery);=20dead-server-safe=20preliminary=20(MainPID=3D0=20gu?= =?UTF-8?q?arded);=20liveness=20+=20generous=20overridable=20stall=20cap?= =?UTF-8?q?=20with=20journal=20diagnosis=20on=20expiry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 71 ++++++++++++------- 1 file changed, 45 insertions(+), 26 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 898fd3754..75dca9cdb 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -111,8 +111,14 @@ RUN=$(mktemp -d) # private per-run dir for the survival snapshots docker ps -q --filter name=incarnation | xargs -r docker inspect \ --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort | tee "$RUN/souls-before.txt" CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) +# The server may be DEAD at this point (this procedure is also the +# recovery path for a dead managed server): MainPID=0 then, and reading +# /proc/0/environ would abort under set -e — guard it: SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) -TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p') +TOK="" +if [ "${SPID:-0}" -gt 0 ] 2>/dev/null; then + TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p' || true) +fi if [ -z "$TOK" ]; then TOK=$(python3 - "$CHECKOUT/.env" <<'PY' import re,sys @@ -132,7 +138,7 @@ fi test -n "$TOK" || { echo "AUTH_TOKEN not found — extract it manually"; exit 1; } umask 077 printf 'x-auth-token: %s\n' "$TOK" > "$RUN/hdr" -trap 'rm -f "$RUN/hdr"' EXIT +trap 'rm -f "${RUN:-/nonexistent}/hdr"' EXIT # ${RUN:-} survives set -u if curl -s -o /dev/null -w '%{http_code}' -H @"$RUN/hdr" \ http://127.0.0.1:3001/api/runtime/readiness | grep -q 200; then curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ @@ -177,12 +183,14 @@ the procedure mid-way and leave the service down. Two sequences: ```bash systemctl --user start freshell-supervisor.service SOCK="$HOME/.freshell/managed-runtime/control/supervisor.sock" - # The supervisor binds its socket only AFTER startup reconciliation, and - # reconciliation sizing is CONFIGURABLE (FRESHELL_RUNTIME_STARTUP_CONCURRENCY, - # per-soul timeout up to 900s, plus pending loss cleanup) — so no fixed - # formula can bound it reliably. Wait on supervisor-unit LIVENESS as the - # failure detector instead: a healthy-but-slow supervisor is not a - # failure, a dead unit is: + # Two failure detectors: supervisor-unit LIVENESS (fast — a dead unit + # ends the wait immediately) and a generous absolute CAP (stall — a + # live-but-wedged process must not hang the redeploy forever). + # Reconciliation sizing is configurable, so no formula is exact; the + # 3600s default exceeds any sane inventory, and operators with extreme + # configs can raise it: + CAP=${FRESHELL_PROBE_SOCKET_WAIT_SECONDS:-3600} + WAITED=0 while true; do # CONNECT-probe, not a file test: a stopped supervisor leaves its old # socket FILE on disk, so [ -S ... ] passes on a dead listener: @@ -194,19 +202,22 @@ except OSError: sys.exit(1)' "$SOCK" 2>/dev/null; then fi systemctl --user is-active freshell-supervisor.service >/dev/null \ || { echo "supervisor unit is not active"; exit 1; } - sleep 5 + [ "$WAITED" -lt "$CAP" ] || { + echo "supervisor alive but socket not bound after ${WAITED}s — check its journal:" + journalctl --user -u freshell-supervisor.service -n 30 --no-pager + echo "if your config legitimately needs longer, raise FRESHELL_PROBE_SOCKET_WAIT_SECONDS" + exit 1 + } + WAITED=$((WAITED + 5)); sleep 5 done systemctl --user start freshell-rust.service ``` - The connect-probe waits for a LIVE listener (a stale file fails the - connect) and the unit-liveness check is the real failure detector — the - loop ends when the socket appears or the supervisor dies, whatever the - reconciliation duration. If the server unit fails its first start in a - race window, that is this socket ordering, not a bad build — wait for - the live socket and start it again (repeated immediate failures can - also trip the unit's start limit). No soul can be created during the - build. This is a full outage for the duration of the build. + If the server unit fails its first start in a race window, that is this + socket ordering, not a bad build — wait for the live socket and start + it again (repeated immediate failures can also trip the unit's start + limit). No soul can be created during the build. This is a full outage + for the duration of the build. - The server's soul controller is strictly opt-in (`default = []` in `crates/freshell-server/Cargo.toml`). Setting `FRESHELL_MANAGED_RUNTIME_V1=1` @@ -411,11 +422,14 @@ print("ok:", sys.argv[1])' "$SID" # following the server's dotenv rules for the common forms (fully # quoted values, quoted-then-comment, unquoted with a # whitespace-preceded `#` comment; `v#x` keeps the `#` literally). - # NOTE: /proc//environ does NOT contain variables dotenvy loaded - # at runtime — that is exactly why the .env fallback exists. - SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) - TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p') - if [ -z "$TOK" ]; then + # NOTE: /proc//environ does NOT contain variables dotenvy loaded + # at runtime — that is exactly why the .env fallback exists. + SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) + TOK="" + if [ "${SPID:-0}" -gt 0 ] 2>/dev/null; then + TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p' || true) + fi + if [ -z "$TOK" ]; then TOK=$(python3 - "$CHECKOUT/.env" <<'PY' import re,sys last=None @@ -435,8 +449,10 @@ PY umask 077 printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" # header file, not argv # ONE EXIT trap per shell: this REPLACES the preliminary capture's - # trap, so scrub BOTH token headers here or $RUN/hdr would leak: - trap 'rm -f "$RUN/hdr" "$D/hdr"' EXIT + # trap, so scrub BOTH token headers here or $RUN/hdr would leak (the + # ${VAR:-} defaults keep the trap functional under set -u even in a + # lost-shell recovery where RUN is unset): + trap 'rm -f "${RUN:-/nonexistent}/hdr" "${D:-/nonexistent}/hdr"' EXIT printf 'D=%q\nWS=%q\nCHECKOUT=%q\n' "$D" "$WS" "$CHECKOUT" > "$D/run.env" # create the probe pane (example); record terminalId from the response: curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ @@ -585,7 +601,10 @@ print("UI pane is managed:", hits[0]["soulId"])' "$BWS" # hard non-empty assert): if [ ! -f "$D/hdr" ]; then SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) - TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p') + TOK="" + if [ "${SPID:-0}" -gt 0 ] 2>/dev/null; then + TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p' || true) + fi if [ -z "$TOK" ]; then TOK=$(python3 - "$CHECKOUT/.env" <<'PY' import re,sys @@ -605,7 +624,7 @@ PY test -n "$TOK" || { echo "AUTH_TOKEN not found — extract it manually"; exit 1; } umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" fi - trap 'rm -f "$RUN/hdr" "$D/hdr"' EXIT # BOTH token headers, every exit path + trap 'rm -f "${RUN:-/nonexistent}/hdr" "${D:-/nonexistent}/hdr"' EXIT # BOTH headers, every exit path, set -u-safe if [ -z "${SOUL_ID:-}" ]; then # Recovery mode: derive the probe soul from the unique workspace — # no independent terminalId check applies here; WS uniqueness IS From 4b338a826248dcc33809bc226d46f3836d4bee5a Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 12:44:29 -0700 Subject: [PATCH 26/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2024?= =?UTF-8?q?=20=E2=80=94=20dead-server=20token=20acquisition=20deferred=20t?= =?UTF-8?q?o=20step=205=20(no=20abort=20when=20only=20EnvironmentFiles=20h?= =?UTF-8?q?old=20it);=20stall=20cap=20sized=20from=20the=20captured=20inve?= =?UTF-8?q?ntory=20(900s/soul=20at=204-way,=20doubled,=203600s=20floor)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 36 +++++++++++++------ 1 file changed, 25 insertions(+), 11 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 75dca9cdb..870685348 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -121,9 +121,12 @@ if [ "${SPID:-0}" -gt 0 ] 2>/dev/null; then fi if [ -z "$TOK" ]; then TOK=$(python3 - "$CHECKOUT/.env" <<'PY' -import re,sys +import os,re,sys +path=sys.argv[1] +if not os.path.exists(path): sys.exit(0) # no .env — token comes from + # the unit's EnvironmentFile last=None -for line in open(sys.argv[1]): +for line in open(path): m=re.match(r'^\s*(?:export\s+)?AUTH_TOKEN\s*=\s*(.*)$', line.rstrip("\n")) if m: last=m.group(1) if last is None: sys.exit(0) @@ -135,11 +138,18 @@ else: PY ) fi -test -n "$TOK" || { echo "AUTH_TOKEN not found — extract it manually"; exit 1; } umask 077 -printf 'x-auth-token: %s\n' "$TOK" > "$RUN/hdr" +if [ -n "$TOK" ]; then + printf 'x-auth-token: %s\n' "$TOK" > "$RUN/hdr" +else + # Dead server + token only in the unit's EnvironmentFiles: DEFER token + # acquisition to step 5 (the server is alive by then) rather than + # aborting — the API-less capture below still records the containers: + echo "AUTH_TOKEN unresolvable while the server is down — container-count capture only; step 5 re-acquires the token" >&2 + : > "$RUN/desired-running.txt" +fi trap 'rm -f "${RUN:-/nonexistent}/hdr"' EXIT # ${RUN:-} survives set -u -if curl -s -o /dev/null -w '%{http_code}' -H @"$RUN/hdr" \ +if [ -f "$RUN/hdr" ] && curl -s -o /dev/null -w '%{http_code}' -H @"$RUN/hdr" \ http://127.0.0.1:3001/api/runtime/readiness | grep -q 200; then curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys @@ -184,12 +194,16 @@ the procedure mid-way and leave the service down. Two sequences: systemctl --user start freshell-supervisor.service SOCK="$HOME/.freshell/managed-runtime/control/supervisor.sock" # Two failure detectors: supervisor-unit LIVENESS (fast — a dead unit - # ends the wait immediately) and a generous absolute CAP (stall — a - # live-but-wedged process must not hang the redeploy forever). - # Reconciliation sizing is configurable, so no formula is exact; the - # 3600s default exceeds any sane inventory, and operators with extreme - # configs can raise it: - CAP=${FRESHELL_PROBE_SOCKET_WAIT_SECONDS:-3600} + # ends the wait immediately) and an inventory-sized absolute CAP + # (stall — a live-but-wedged process must not hang the redeploy + # forever). Worst-case reconciliation is 900s per soul at four-way + # concurrency (225s effective each); double that for margin and floor + # at an hour. Operators with extreme configs can override the env var: + N=$(wc -l < "$RUN/desired-running.txt" 2>/dev/null || echo 0) + [ "$N" -gt 0 ] || N=$(wc -l < "$RUN/souls-before.txt" 2>/dev/null || echo 0) + WC=$(( N * 225 * 2 )) + [ "$WC" -ge 3600 ] || WC=3600 + CAP=${FRESHELL_PROBE_SOCKET_WAIT_SECONDS:-$WC} WAITED=0 while true; do # CONNECT-probe, not a file test: a stopped supervisor leaves its old From 2cfeae0f21d6f93b81cb150f0b1d82a084e010b3 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 12:54:25 -0700 Subject: [PATCH 27/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2025?= =?UTF-8?q?=20=E2=80=94=20step-4=20refresh=20re-acquires=20the=20token=20p?= =?UTF-8?q?ost-deploy;=20API-less=20fallback=20limitation=20stated=20hones?= =?UTF-8?q?tly;=20all=20workspace=20lookups=20assert=20exactly=20one=20mat?= =?UTF-8?q?ch=20(one=20record=20per=20soul,=20refuse=20on=20ambiguity)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 40 +++++++++++++++---- 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 870685348..ae79bf44d 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -375,7 +375,19 @@ drop-in, plus `freshell-supervisor.service`): ```bash # If the preliminary capture fell back to an empty list (featureless # or stopped server), refresh it NOW from the freshly deployed API — - # an empty sweep would otherwise pass vacuously: + # an empty sweep would otherwise pass vacuously. The server is alive + # at this point, so if the deferred-token path left no $RUN/hdr, + # re-acquire the token here exactly as the preliminary step does: + if [ ! -f "$RUN/hdr" ]; then + SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) + TOK="" + if [ "${SPID:-0}" -gt 0 ] 2>/dev/null; then + TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p' || true) + fi + test -n "$TOK" \ + || { echo "AUTH_TOKEN unresolvable even with the server running — extract it manually"; exit 1; } + umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$RUN/hdr" + fi if [ ! -s "$RUN/desired-running.txt" ]; then curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys @@ -401,7 +413,13 @@ print("ok:", sys.argv[1])' "$SID" needs the supervisor journal and that soul's record investigated before the deploy is called good. A zero-soul sweep after a SUCCESSFUL refresh is a legitimate empty instance — but the printed - count makes that visible instead of vacuous. + count makes that visible instead of vacuous. HONEST LIMITATION of the + API-less fallback: with no pre-restart API, no soul-ID baseline + exists — the post-restart list is compared against nothing, so this + sweep cannot detect a registry swap (old containers surviving while + the new registry reports zero souls). The container-level survival + check still applies; if a registry change is even plausible in your + deploy, re-capture and re-verify after the server is up. 5. New-soul probe: create a terminal pane in a real workspace that is NOT the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits @@ -558,8 +576,11 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] -hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1] and s.get("launchState")=="running"] -assert hits, "no managed soul for the UI pane — client is stale or not negotiating" +hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] +# EXACTLY one soul per probe workspace; two hits means this workspace was +# reused by an earlier attempt — refuse rather than pick arbitrarily: +assert len(hits)==1, ("workspace ambiguous (mint a fresh BWS and retry)", hits) +assert hits[0].get("launchState")=="running", "soul not running — client took the host-side route?" print("UI pane is managed:", hits[0]["soulId"])' "$BWS" ``` @@ -654,8 +675,11 @@ PY | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -assert hits, "no soul for this workspace — nothing to stop" -s=hits[-1] # newest incarnation (inventory is oldest-first) +# The API returns ONE record per soul (its current incarnation), so a +# unique probe workspace must match exactly one record; ambiguity means +# workspace reuse — refuse rather than pick arbitrarily: +assert len(hits)==1, ("workspace ambiguous — do not stop anything; resolve manually", hits) +s=hits[0] print(s["soulId"], s["intentRevision"], s["terminalId"])' "$WS") read -r SOUL_ID REV TID <<< "$PROBE" else @@ -689,8 +713,10 @@ print("stopped:", o["outcome"])' | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] +# exactly one soul per probe workspace; refuse on ambiguity: +assert len(hits)<=1, ("workspace ambiguous — do not stop anything; resolve manually", hits) if hits: - s=hits[-1] + s=hits[0] print(s["soulId"], s["intentRevision"], s["launchState"]) else: print("")' "$BWS") From b654e642141ec90be223afaeb399fd6bebc108d2 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 13:03:29 -0700 Subject: [PATCH 28/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2026?= =?UTF-8?q?=20=E2=80=94=20lookups=20group=20by=20soulId=20and=20act=20on?= =?UTF-8?q?=20the=20newest=20incarnation=20row=20(inventory=20is=20per-inc?= =?UTF-8?q?arnation);=20cap=20sized=20from=20effective=20FRESHELL=5FRUNTIM?= =?UTF-8?q?E=5FSTARTUP=5FCONCURRENCY;=20positive=20orphan=20check:=20basel?= =?UTF-8?q?ine=20containers'=20souls=20must=20exist=20in=20the=20deployed?= =?UTF-8?q?=20registry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 75 +++++++++++++------ 1 file changed, 54 insertions(+), 21 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index ae79bf44d..f5951e9fc 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -196,12 +196,21 @@ the procedure mid-way and leave the service down. Two sequences: # Two failure detectors: supervisor-unit LIVENESS (fast — a dead unit # ends the wait immediately) and an inventory-sized absolute CAP # (stall — a live-but-wedged process must not hang the redeploy - # forever). Worst-case reconciliation is 900s per soul at four-way - # concurrency (225s effective each); double that for margin and floor - # at an hour. Operators with extreme configs can override the env var: + # forever). Size from the ACTUAL effective concurrency: read + # FRESHELL_RUNTIME_STARTUP_CONCURRENCY from the supervisor unit's + # environment files (default 4), and use the documented 900s per-soul + # reconciliation ceiling: worst-case N*900/C seconds, doubled for + # margin, floored at an hour. Extreme configs can override the env var: + C=4 + for EF in $(systemctl --user show freshell-supervisor.service \ + -p EnvironmentFiles --value | tr ' ' '\n' \ + | sed 's/(.*//' | grep -v '^$'); do + V=$(sed -n 's/^FRESHELL_RUNTIME_STARTUP_CONCURRENCY=//p' "$EF" 2>/dev/null | tail -1) + [ -n "$V" ] && C="$V" + done N=$(wc -l < "$RUN/desired-running.txt" 2>/dev/null || echo 0) [ "$N" -gt 0 ] || N=$(wc -l < "$RUN/souls-before.txt" 2>/dev/null || echo 0) - WC=$(( N * 225 * 2 )) + WC=$(( N * 900 / C * 2 )) [ "$WC" -ge 3600 ] || WC=3600 CAP=${FRESHELL_PROBE_SOCKET_WAIT_SECONDS:-$WC} WAITED=0 @@ -392,9 +401,26 @@ drop-in, plus `freshell-supervisor.service`): curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] +for s in souls: + print(s["soulId"])' > "$RUN/souls-all.txt" + curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys +souls=json.load(sys.stdin)["souls"] for s in souls: if s.get("desiredState")=="running": print(s["soulId"])' > "$RUN/desired-running.txt" + # POSITIVE orphan check for the API-less fallback: every baseline + # container's soul must exist in the DEPLOYED registry — a registry + # swap leaves the containers running but owned by nothing: + while read -r CIDLINE; do + CID=${CIDLINE%% *} + [ -n "$CID" ] || continue + SID=$(docker inspect "$CID" \ + --format '{{index .Config.Labels "com.freshell.soul-id"}}' 2>/dev/null || true) + [ -n "$SID" ] || continue + grep -qx "$SID" "$RUN/souls-all.txt" \ + || { echo "ORPHANED container — soul $SID not in the deployed registry: $CID"; exit 1; } + done < "$RUN/souls-before.txt" fi echo "sweeping $(wc -l < "$RUN/desired-running.txt") desired-running soul(s)" while read -r SID; do @@ -576,12 +602,16 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] -hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -# EXACTLY one soul per probe workspace; two hits means this workspace was -# reused by an earlier attempt — refuse rather than pick arbitrarily: -assert len(hits)==1, ("workspace ambiguous (mint a fresh BWS and retry)", hits) -assert hits[0].get("launchState")=="running", "soul not running — client took the host-side route?" -print("UI pane is managed:", hits[0]["soulId"])' "$BWS" +rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] +# The inventory returns ONE ROW PER INCARNATION (recovered souls keep +# their old rows); the invariant is exactly one SOUL per probe +# workspace. Group by soulId, refuse on ambiguity, and inspect the +# NEWEST incarnation row (the list is creation-ordered): +sids={r["soulId"] for r in rows} +assert len(sids)==1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) +cur=rows[-1] +assert cur.get("launchState")=="running", "soul not running — client took the host-side route?" +print("UI pane is managed:", cur["soulId"])' "$BWS" ``` If the assertion fails, the client is stale or not negotiating @@ -674,12 +704,13 @@ PY PROBE=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] -hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -# The API returns ONE record per soul (its current incarnation), so a -# unique probe workspace must match exactly one record; ambiguity means -# workspace reuse — refuse rather than pick arbitrarily: -assert len(hits)==1, ("workspace ambiguous — do not stop anything; resolve manually", hits) -s=hits[0] +rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] +# One row per INCARNATION; the invariant is exactly one SOUL per probe +# workspace — group by soulId, refuse on ambiguity, act on the NEWEST +# incarnation row (creation-ordered list): +sids={r["soulId"] for r in rows} +assert len(sids)==1, ("workspace ambiguous — do not stop anything; resolve manually", rows) +s=rows[-1] print(s["soulId"], s["intentRevision"], s["terminalId"])' "$WS") read -r SOUL_ID REV TID <<< "$PROBE" else @@ -712,11 +743,13 @@ print("stopped:", o["outcome"])' BS=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] -hits=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -# exactly one soul per probe workspace; refuse on ambiguity: -assert len(hits)<=1, ("workspace ambiguous — do not stop anything; resolve manually", hits) -if hits: - s=hits[0] +rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] +# One row per INCARNATION; exactly one SOUL per probe workspace; act on +# the NEWEST incarnation row (creation-ordered list): +sids={r["soulId"] for r in rows} +assert len(sids)<=1, ("workspace ambiguous — do not stop anything; resolve manually", rows) +if rows: + s=rows[-1] print(s["soulId"], s["intentRevision"], s["launchState"]) else: print("")' "$BWS") From 6cf1db01dc5f3f8996582e94ccf7b8f7b44fe5be Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 13:17:00 -0700 Subject: [PATCH 29/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2027?= =?UTF-8?q?=20=E2=80=94=20set-e-safe=20concurrency=20read=20(if-forms,=20E?= =?UTF-8?q?nvironment=3D=20lines,=20supervisor=201..16=20clamp);=20install?= =?UTF-8?q?ation-scoped=20docker=20filters;=20orphan=20check=20matches=20(?= =?UTF-8?q?soul,=20container)=20pairs=20with=20surfaced=20inspect=20failur?= =?UTF-8?q?es;=20API=20data-model=20comment=20corrected=20(collapses=20to?= =?UTF-8?q?=20latest=20row=20per=20soul)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 77 ++++++++++++------- 1 file changed, 50 insertions(+), 27 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index f5951e9fc..80217e61c 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -108,7 +108,16 @@ post-deploy soul sweep: ```bash set -euo pipefail RUN=$(mktemp -d) # private per-run dir for the survival snapshots -docker ps -q --filter name=incarnation | xargs -r docker inspect \ +# Scope the capture to THIS installation: the daemon may host containers +# from other Freshell installations with the same name shape, and the +# installation label is the only discriminator: +INSTALL_ID=$(sqlite3 "$HOME/.freshell/managed-runtime/registry/runtime.sqlite3" \ + "select installation_id from installation") +test -n "$INSTALL_ID" || { echo "cannot read installation id from the registry"; exit 1; } +echo "$INSTALL_ID" > "$RUN/installation-id" +docker ps -q --filter name=incarnation \ + --filter label=com.freshell.installation-id="$INSTALL_ID" \ + | xargs -r docker inspect \ --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort | tee "$RUN/souls-before.txt" CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) # The server may be DEAD at this point (this procedure is also the @@ -196,18 +205,29 @@ the procedure mid-way and leave the service down. Two sequences: # Two failure detectors: supervisor-unit LIVENESS (fast — a dead unit # ends the wait immediately) and an inventory-sized absolute CAP # (stall — a live-but-wedged process must not hang the redeploy - # forever). Size from the ACTUAL effective concurrency: read - # FRESHELL_RUNTIME_STARTUP_CONCURRENCY from the supervisor unit's - # environment files (default 4), and use the documented 900s per-soul - # reconciliation ceiling: worst-case N*900/C seconds, doubled for - # margin, floored at an hour. Extreme configs can override the env var: - C=4 - for EF in $(systemctl --user show freshell-supervisor.service \ - -p EnvironmentFiles --value | tr ' ' '\n' \ - | sed 's/(.*//' | grep -v '^$'); do - V=$(sed -n 's/^FRESHELL_RUNTIME_STARTUP_CONCURRENCY=//p' "$EF" 2>/dev/null | tail -1) - [ -n "$V" ] && C="$V" - done + # forever). Read the EFFECTIVE concurrency the way the supervisor + # does: unit Environment= lines and its EnvironmentFiles, then the + # supervisor's own 1..16 clamp (0 or garbage become 1/defaults, never + # a division by zero). Worst case is N * 900s / C, doubled for + # margin, floored at an hour; extreme configs can override the env + # var. The if-forms below are load-bearing under set -e: an + # `&&`-short-circuit as the loop's last command would make the LOOP + # return 1 and abort the whole sequence: + C="" + V=$(systemctl --user show freshell-supervisor.service -p Environment --value 2>/dev/null \ + | tr ' ' '\n' | sed -n 's/^FRESHELL_RUNTIME_STARTUP_CONCURRENCY=//p' | tail -1) + if [ -n "$V" ]; then C="$V"; fi + if [ -z "$C" ]; then + for EF in $(systemctl --user show freshell-supervisor.service \ + -p EnvironmentFiles --value | tr ' ' '\n' \ + | sed 's/(.*//' | grep -v '^$'); do + V=$(sed -n 's/^FRESHELL_RUNTIME_STARTUP_CONCURRENCY=//p' "$EF" 2>/dev/null | tail -1) + if [ -n "$V" ]; then C="$V"; fi + done + fi + case "$C" in ''|*[!0-9]*) C=4;; esac # absent/garbage -> supervisor default + [ "$C" -ge 1 ] 2>/dev/null || C=1 # supervisor clamps to 1..16 — mirror it + [ "$C" -le 16 ] 2>/dev/null || C=16 N=$(wc -l < "$RUN/desired-running.txt" 2>/dev/null || echo 0) [ "$N" -gt 0 ] || N=$(wc -l < "$RUN/souls-before.txt" 2>/dev/null || echo 0) WC=$(( N * 900 / C * 2 )) @@ -402,7 +422,7 @@ drop-in, plus `freshell-supervisor.service`): | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] for s in souls: - print(s["soulId"])' > "$RUN/souls-all.txt" + print(s["soulId"], s.get("containerId") or "")' > "$RUN/souls-all.txt" curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] @@ -416,10 +436,12 @@ for s in souls: CID=${CIDLINE%% *} [ -n "$CID" ] || continue SID=$(docker inspect "$CID" \ - --format '{{index .Config.Labels "com.freshell.soul-id"}}' 2>/dev/null || true) - [ -n "$SID" ] || continue - grep -qx "$SID" "$RUN/souls-all.txt" \ - || { echo "ORPHANED container — soul $SID not in the deployed registry: $CID"; exit 1; } + --format '{{index .Config.Labels "com.freshell.soul-id"}}') \ + || { echo "docker inspect failed for baseline container $CID"; exit 1; } + # Ownership is per (soul, incarnation, container): match the + # PAIR against the deployed registry, not the soul id alone: + grep -q "^$SID $CID$" "$RUN/souls-all.txt" \ + || { echo "ORPHANED container — ($SID, $CID) not owned by the deployed registry"; exit 1; } done < "$RUN/souls-before.txt" fi echo "sweeping $(wc -l < "$RUN/desired-running.txt") desired-running soul(s)" @@ -603,10 +625,10 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -# The inventory returns ONE ROW PER INCARNATION (recovered souls keep -# their old rows); the invariant is exactly one SOUL per probe -# workspace. Group by soulId, refuse on ambiguity, and inspect the -# NEWEST incarnation row (the list is creation-ordered): +# The registry stores one row per incarnation, but the API collapses to +# the LATEST row per soul — so normally one row per soul. The grouping +# below stays robust either way: exactly one SOUL must match the probe +# workspace; refuse on ambiguity and act on the last matching row: sids={r["soulId"] for r in rows} assert len(sids)==1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) cur=rows[-1] @@ -705,9 +727,9 @@ PY | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -# One row per INCARNATION; the invariant is exactly one SOUL per probe -# workspace — group by soulId, refuse on ambiguity, act on the NEWEST -# incarnation row (creation-ordered list): +# The API collapses per-incarnation rows to the latest per soul; the +# grouping stays robust either way — exactly one SOUL per probe +# workspace, refuse on ambiguity, act on the last matching row: sids={r["soulId"] for r in rows} assert len(sids)==1, ("workspace ambiguous — do not stop anything; resolve manually", rows) s=rows[-1] @@ -744,8 +766,9 @@ print("stopped:", o["outcome"])' | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -# One row per INCARNATION; exactly one SOUL per probe workspace; act on -# the NEWEST incarnation row (creation-ordered list): +# The API collapses per-incarnation rows to the latest per soul; the +# grouping stays robust either way — exactly one SOUL per probe +# workspace, act on the last matching row: sids={r["soulId"] for r in rows} assert len(sids)<=1, ("workspace ambiguous — do not stop anything; resolve manually", rows) if rows: From f9e308935d59b7f1828eaaf3b40fd9202c0a66d1 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 13:26:39 -0700 Subject: [PATCH 30/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2028?= =?UTF-8?q?=20=E2=80=94=20systemd=20precedence=20for=20the=20concurrency?= =?UTF-8?q?=20read=20(inherited=20env,=20Environment=3D,=20then=20Environm?= =?UTF-8?q?entFile=20which=20wins);=20DOCKER=5FHOST=20pinned=20to=20the=20?= =?UTF-8?q?supervisor's=20daemon=20socket;=20executable=20UI=20path=20for?= =?UTF-8?q?=20the=20browser=20check=20(coding-CLI=20directory=20step,=20no?= =?UTF-8?q?t=20plain=20shell);=20after-capture=20installation=20filter=20a?= =?UTF-8?q?ctually=20applied?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 50 ++++++++++++------- 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 80217e61c..5098d9eb3 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -108,6 +108,11 @@ post-deploy soul sweep: ```bash set -euo pipefail RUN=$(mktemp -d) # private per-run dir for the survival snapshots +# Pin the Docker CLI to the SAME daemon socket the supervisor uses (its +# unit passes --docker-socket %t/docker.sock, the user runtime dir). A +# machine with a second Docker daemon would otherwise capture against +# the wrong one and the survival check would pass vacuously: +export DOCKER_HOST="unix://${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/docker.sock" # Scope the capture to THIS installation: the daemon may host containers # from other Freshell installations with the same name shape, and the # installation label is the only discriminator: @@ -205,26 +210,28 @@ the procedure mid-way and leave the service down. Two sequences: # Two failure detectors: supervisor-unit LIVENESS (fast — a dead unit # ends the wait immediately) and an inventory-sized absolute CAP # (stall — a live-but-wedged process must not hang the redeploy - # forever). Read the EFFECTIVE concurrency the way the supervisor - # does: unit Environment= lines and its EnvironmentFiles, then the - # supervisor's own 1..16 clamp (0 or garbage become 1/defaults, never - # a division by zero). Worst case is N * 900s / C, doubled for - # margin, floored at an hour; extreme configs can override the env - # var. The if-forms below are load-bearing under set -e: an - # `&&`-short-circuit as the loop's last command would make the LOOP - # return 1 and abort the whole sequence: + # forever). Read the EFFECTIVE concurrency in systemd's own precedence + # order — the user manager's inherited environment first, then the + # unit's Environment= lines, then its EnvironmentFile= values (systemd + # applies the FILE later, so it WINS) — then mirror the supervisor's + # own 1..16 clamp (0 or garbage become 1/defaults, never a division + # by zero). Worst case is N * 900s / C, doubled for margin, floored at + # an hour; extreme configs can override the env var. The if-forms are + # load-bearing under set -e: an `&&`-short-circuit as the loop's last + # command would make the LOOP return 1 and abort the whole sequence: C="" + V=$(systemctl --user show-environment 2>/dev/null \ + | sed -n 's/^FRESHELL_RUNTIME_STARTUP_CONCURRENCY=//p' | tail -1) + if [ -n "$V" ]; then C="$V"; fi V=$(systemctl --user show freshell-supervisor.service -p Environment --value 2>/dev/null \ | tr ' ' '\n' | sed -n 's/^FRESHELL_RUNTIME_STARTUP_CONCURRENCY=//p' | tail -1) if [ -n "$V" ]; then C="$V"; fi - if [ -z "$C" ]; then - for EF in $(systemctl --user show freshell-supervisor.service \ - -p EnvironmentFiles --value | tr ' ' '\n' \ - | sed 's/(.*//' | grep -v '^$'); do - V=$(sed -n 's/^FRESHELL_RUNTIME_STARTUP_CONCURRENCY=//p' "$EF" 2>/dev/null | tail -1) - if [ -n "$V" ]; then C="$V"; fi - done - fi + for EF in $(systemctl --user show freshell-supervisor.service \ + -p EnvironmentFiles --value | tr ' ' '\n' \ + | sed 's/(.*//' | grep -v '^$'); do + V=$(sed -n 's/^FRESHELL_RUNTIME_STARTUP_CONCURRENCY=//p' "$EF" 2>/dev/null | tail -1) + if [ -n "$V" ]; then C="$V"; fi + done case "$C" in ''|*[!0-9]*) C=4;; esac # absent/garbage -> supervisor default [ "$C" -ge 1 ] 2>/dev/null || C=1 # supervisor clamps to 1..16 — mirror it [ "$C" -le 16 ] 2>/dev/null || C=16 @@ -380,7 +387,9 @@ drop-in, plus `freshell-supervisor.service`): overwrite the baseline and compare the file against itself: ```bash - docker ps -q --filter name=incarnation | xargs -r docker inspect \ + docker ps -q --filter name=incarnation \ + --filter label=com.freshell.installation-id="$(cat "$RUN/installation-id")" \ + | xargs -r docker inspect \ --format '{{.Id}} {{.Name}} {{.State.StartedAt}}' | sort > "$RUN/souls-after.txt" LOST=$(comm -23 "$RUN/souls-before.txt" "$RUN/souls-after.txt") test -z "$LOST" || { echo "SOULS LOST/RESTARTED:"; echo "$LOST"; exit 1; } @@ -620,7 +629,12 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T BWS=$(mktemp -d "$PROBE_ROOT/bws.XXXXXX") # unique per run chmod 0770 "$BWS" # same in-soul-shell access requirement as $WS echo "BWS=$BWS" >> "$D/run.env" - # ...hard-refresh the web client, create a terminal pane at $BWS... + # Hard-refresh the web client, then in the UI: open the pane picker + # and choose a coding-CLI provider (or a fresh-agent type) — NOT a + # plain shell: shell panes skip the directory step and always take + # the default workspace, which would fail this check for the wrong + # reason. The provider's directory step has a free-text PATH INPUT: + # type the absolute $BWS path there and confirm. Then: curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] From 91d2db5a77c93b722fd6e6185469b21080c71a69 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 13:37:44 -0700 Subject: [PATCH 31/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2029?= =?UTF-8?q?=20=E2=80=94=20DOCKER=5FCONTEXT=20unset=20(it=20overrides=20DOC?= =?UTF-8?q?KER=5FHOST);=20step-4=20always=20re-acquires=20the=20live=20tok?= =?UTF-8?q?en=20and=20rewrites=20the=20header;=20ownership=20check=20runs?= =?UTF-8?q?=20in=20every=20path=20and=20requires=20a=20running=20(soul,=20?= =?UTF-8?q?container)=20row;=20browser=20check=20pinned=20to=20OpenCode=20?= =?UTF-8?q?(the=20only=20managed-enabled=20CLI=20provider);=20no-container?= =?UTF-8?q?=20probe=20souls=20handled=20honestly?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 112 +++++++++--------- 1 file changed, 57 insertions(+), 55 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 5098d9eb3..af3eb7696 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -111,7 +111,9 @@ RUN=$(mktemp -d) # private per-run dir for the survival snapshots # Pin the Docker CLI to the SAME daemon socket the supervisor uses (its # unit passes --docker-socket %t/docker.sock, the user runtime dir). A # machine with a second Docker daemon would otherwise capture against -# the wrong one and the survival check would pass vacuously: +# the wrong one and the survival check would pass vacuously. DOCKER_CONTEXT +# must go too — it takes precedence over DOCKER_HOST: +unset DOCKER_CONTEXT || true export DOCKER_HOST="unix://${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/docker.sock" # Scope the capture to THIS installation: the daemon may host containers # from other Freshell installations with the same name shape, and the @@ -411,48 +413,45 @@ drop-in, plus `freshell-supervisor.service`): sees): ```bash - # If the preliminary capture fell back to an empty list (featureless - # or stopped server), refresh it NOW from the freshly deployed API — - # an empty sweep would otherwise pass vacuously. The server is alive - # at this point, so if the deferred-token path left no $RUN/hdr, - # re-acquire the token here exactly as the preliminary step does: - if [ ! -f "$RUN/hdr" ]; then - SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) - TOK="" - if [ "${SPID:-0}" -gt 0 ] 2>/dev/null; then - TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p' || true) - fi - test -n "$TOK" \ - || { echo "AUTH_TOKEN unresolvable even with the server running — extract it manually"; exit 1; } - umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$RUN/hdr" + # The server is ALIVE now, so its environment is the authoritative + # token source — always re-acquire and REWRITE the header (a header + # minted while the server was down may carry a stale .env value that + # the unit's EnvironmentFile actually overrides): + SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) + TOK="" + if [ "${SPID:-0}" -gt 0 ] 2>/dev/null; then + TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p' || true) fi - if [ ! -s "$RUN/desired-running.txt" ]; then - curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ - | python3 -c 'import json,sys + test -n "$TOK" \ + || { echo "AUTH_TOKEN unresolvable even with the server running — extract it manually"; exit 1; } + umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$RUN/hdr" + # Always re-fetch the DEPLOYED registry's soul list (this also fills + # the API-less fallback's empty desired-running baseline — an empty + # sweep would otherwise pass vacuously): + curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] for s in souls: - print(s["soulId"], s.get("containerId") or "")' > "$RUN/souls-all.txt" - curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ - | python3 -c 'import json,sys + print(s["soulId"], s.get("containerId") or "-", s.get("launchState") or "-")' > "$RUN/souls-all.txt" + curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] for s in souls: if s.get("desiredState")=="running": print(s["soulId"])' > "$RUN/desired-running.txt" - # POSITIVE orphan check for the API-less fallback: every baseline - # container's soul must exist in the DEPLOYED registry — a registry - # swap leaves the containers running but owned by nothing: - while read -r CIDLINE; do - CID=${CIDLINE%% *} - [ -n "$CID" ] || continue - SID=$(docker inspect "$CID" \ - --format '{{index .Config.Labels "com.freshell.soul-id"}}') \ - || { echo "docker inspect failed for baseline container $CID"; exit 1; } - # Ownership is per (soul, incarnation, container): match the - # PAIR against the deployed registry, not the soul id alone: - grep -q "^$SID $CID$" "$RUN/souls-all.txt" \ - || { echo "ORPHANED container — ($SID, $CID) not owned by the deployed registry"; exit 1; } - done < "$RUN/souls-before.txt" - fi + # POSITIVE ownership check for EVERY baseline container: its + # (soul, container) pair must appear in the deployed registry with a + # RUNNING incarnation — a registry swap or restore leaves containers + # running but owned by nothing (or by a historical stopped row): + while read -r CIDLINE; do + CID=${CIDLINE%% *} + [ -n "$CID" ] || continue + SID=$(docker inspect "$CID" \ + --format '{{index .Config.Labels "com.freshell.soul-id"}}') \ + || { echo "docker inspect failed for baseline container $CID"; exit 1; } + grep -q "^$SID $CID running$" "$RUN/souls-all.txt" \ + || { echo "ORPHANED container — ($SID, $CID) has no running row in the deployed registry"; exit 1; } + done < "$RUN/souls-before.txt" echo "sweeping $(wc -l < "$RUN/desired-running.txt") desired-running soul(s)" while read -r SID; do [ -n "$SID" ] || continue @@ -468,15 +467,13 @@ print("ok:", sys.argv[1])' "$SID" A blocked or lost `recoveryState` can hide behind a `"running"` `launchState` — that is why both fields are asserted. Any failure here needs the supervisor journal and that soul's record investigated - before the deploy is called good. A zero-soul sweep after a - SUCCESSFUL refresh is a legitimate empty instance — but the printed - count makes that visible instead of vacuous. HONEST LIMITATION of the - API-less fallback: with no pre-restart API, no soul-ID baseline - exists — the post-restart list is compared against nothing, so this - sweep cannot detect a registry swap (old containers surviving while - the new registry reports zero souls). The container-level survival - check still applies; if a registry change is even plausible in your - deploy, re-capture and re-verify after the server is up. + before the deploy is called good. A zero-soul sweep is a legitimate + empty instance — but the printed count makes that visible instead of + vacuous. The earlier API-less-fallback limitation (no pre-restart + soul-ID baseline) is now covered positively: the ownership check + above proves every baseline container has a RUNNING row in the + DEPLOYED registry, which is what detects a registry swap even without + a pre-restart baseline. 5. New-soul probe: create a terminal pane in a real workspace that is NOT the operator home itself (a subdirectory of it is fine) — when the workspace equals the operator home, the supervisor deliberately omits @@ -630,10 +627,13 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T chmod 0770 "$BWS" # same in-soul-shell access requirement as $WS echo "BWS=$BWS" >> "$D/run.env" # Hard-refresh the web client, then in the UI: open the pane picker - # and choose a coding-CLI provider (or a fresh-agent type) — NOT a - # plain shell: shell panes skip the directory step and always take - # the default workspace, which would fail this check for the wrong - # reason. The provider's directory step has a free-text PATH INPUT: + # and choose OPENCODE specifically — the only managed-enabled + # coding-CLI provider in the current release (Claude, Codex, and + # Amplifier are pending live qualification and the fresh-agent modes + # are release-disabled, so choosing those fails this check on a + # correctly deployed client). A plain shell pane will NOT do either: + # shell panes skip the directory step and always take the default + # workspace. OpenCode's directory step has a free-text PATH INPUT: # type the absolute $BWS path there and confirm. Then: curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys @@ -773,9 +773,6 @@ print("stopped:", o["outcome"])' # Look it up by workspace in ANY launchState — a blocked or exited # probe soul still must not be orphaned against a deleted workspace: if [ -n "${BWS:-}" ]; then - # Select the NEWEST record for this workspace (the inventory lists - # incarnations oldest first, so after a recovery the newest entry is - # the live one; an older stopped incarnation may also match): BS=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] @@ -787,12 +784,17 @@ sids={r["soulId"] for r in rows} assert len(sids)<=1, ("workspace ambiguous — do not stop anything; resolve manually", rows) if rows: s=rows[-1] - print(s["soulId"], s["intentRevision"], s["launchState"]) + print(s["soulId"], s["intentRevision"], s["launchState"], s.get("containerId") or "-") else: print("")' "$BWS") if [ -n "$BS" ]; then - read -r BS_ID BS_REV BS_STATE <<< "$BS" - if [ "$BS_STATE" != "stopped" ]; then + read -r BS_ID BS_REV BS_STATE BS_CID <<< "$BS" + if [ "$BS_CID" = "-" ]; then + # A probe that failed before container creation owns no + # container — there is nothing to stop; the supervisor has + # no live handle for it: + echo "browser probe soul has no container — nothing to stop" + elif [ "$BS_STATE" != "stopped" ]; then curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$BS_REV}" \ http://127.0.0.1:3001/api/runtime/souls/$BS_ID/stop \ From 06989808510a8e0c75906f7bd9e51856a6257a90 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 13:45:46 -0700 Subject: [PATCH 32/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2030?= =?UTF-8?q?=20=E2=80=94=20step-4=20token=20re-acquire=20restores=20the=20.?= =?UTF-8?q?env=20dotenv=20fallback=20(dotenvy-loaded=20tokens=20are=20invi?= =?UTF-8?q?sible=20in=20/proc);=20pre-restart=20baseline=20preserved,=20sw?= =?UTF-8?q?eep=20covers=20baseline=20=E2=88=AA=20deployed=20with=20a=20gon?= =?UTF-8?q?e-from-registry=20failure?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 49 ++++++++++++++----- 1 file changed, 37 insertions(+), 12 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index af3eb7696..3127d3c2d 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -413,21 +413,43 @@ drop-in, plus `freshell-supervisor.service`): sees): ```bash - # The server is ALIVE now, so its environment is the authoritative - # token source — always re-acquire and REWRITE the header (a header - # minted while the server was down may carry a stale .env value that - # the unit's EnvironmentFile actually overrides): + # The server is ALIVE now — re-acquire the token (live environment + # first; the .env dotenv fallback second, because a server that + # loads its token from the checkout's .env never exposes it in + # /proc//environ) and REWRITE the header: one minted while the + # server was down may carry a stale .env value the unit's + # EnvironmentFile actually overrides: SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) TOK="" if [ "${SPID:-0}" -gt 0 ] 2>/dev/null; then TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p' || true) fi + if [ -z "$TOK" ]; then + TOK=$(python3 - "$(systemctl --user show freshell-rust.service -p WorkingDirectory --value)/.env" <<'PY' +import os,re,sys +path=sys.argv[1] +if not os.path.exists(path): sys.exit(0) +last=None +for line in open(path): + m=re.match(r'^\s*(?:export\s+)?AUTH_TOKEN\s*=\s*(.*)$', line.rstrip("\n")) + if m: last=m.group(1) +if last is None: sys.exit(0) +if last[:1] in ('"',"'"): + end=last.find(last[0],1) + print(last[1:end] if end>0 else last[1:]) +else: + print(re.split(r'\s#', last.strip(), 1)[0].strip()) +PY +) + fi test -n "$TOK" \ || { echo "AUTH_TOKEN unresolvable even with the server running — extract it manually"; exit 1; } umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$RUN/hdr" - # Always re-fetch the DEPLOYED registry's soul list (this also fills - # the API-less fallback's empty desired-running baseline — an empty - # sweep would otherwise pass vacuously): + # Fetch the DEPLOYED registry's soul list into a SEPARATE file — the + # pre-restart baseline must survive, because a soul that DISAPPEARED + # from the registry is exactly what the sweep must catch (this also + # fills the API-less fallback's empty baseline — an empty sweep + # would otherwise pass vacuously): curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] @@ -438,7 +460,7 @@ for s in souls: souls=json.load(sys.stdin)["souls"] for s in souls: if s.get("desiredState")=="running": - print(s["soulId"])' > "$RUN/desired-running.txt" + print(s["soulId"])' > "$RUN/desired-running-after.txt" # POSITIVE ownership check for EVERY baseline container: its # (soul, container) pair must appear in the deployed registry with a # RUNNING incarnation — a registry swap or restore leaves containers @@ -452,16 +474,19 @@ for s in souls: grep -q "^$SID $CID running$" "$RUN/souls-all.txt" \ || { echo "ORPHANED container — ($SID, $CID) has no running row in the deployed registry"; exit 1; } done < "$RUN/souls-before.txt" - echo "sweeping $(wc -l < "$RUN/desired-running.txt") desired-running soul(s)" - while read -r SID; do + echo "sweeping $(sort -u "$RUN/desired-running.txt" "$RUN/desired-running-after.txt" | wc -l) desired-running soul(s) (baseline ∪ deployed)" + sort -u "$RUN/desired-running.txt" "$RUN/desired-running-after.txt" \ + | while read -r SID; do [ -n "$SID" ] || continue curl -s -H @"$RUN/hdr" "http://127.0.0.1:3001/api/runtime/souls/$SID" \ | python3 -c 'import json,sys -s=json.load(sys.stdin)["soul"] +o=json.load(sys.stdin) +s=o.get("soul") +assert s is not None, ("SOUL GONE from the deployed registry", sys.argv[1], o) ls=s.get("launchState"); rs=s.get("recoveryState") assert ls=="running" and rs not in ("blocked","lost"), (sys.argv[1], ls, rs) print("ok:", sys.argv[1])' "$SID" - done < "$RUN/desired-running.txt" + done ``` A blocked or lost `recoveryState` can hide behind a `"running"` From e9f6c5cbcdab4615c219f66a6d92008f6cbbb66d Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 13:55:05 -0700 Subject: [PATCH 33/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2031?= =?UTF-8?q?=20=E2=80=94=20dead-server=20baseline=20read=20from=20the=20reg?= =?UTF-8?q?istry=20sqlite=20(desired=5Fstate=20is=20readable=20with=20the?= =?UTF-8?q?=20server=20down);=20BWS=20cleanup=20requires=20positive=20veri?= =?UTF-8?q?fied-stopped=20identification=20before=20workspace=20deletion;?= =?UTF-8?q?=20data-model=20comments=20made=20interpretation-agnostic?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 52 ++++++++++++------- 1 file changed, 32 insertions(+), 20 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 3127d3c2d..ac69abc1f 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -174,8 +174,15 @@ for s in souls: if s.get("desiredState")=="running": print(s["soulId"])' > "$RUN/desired-running.txt" else - echo "runtime API unavailable (featureless or stopped server) — container count only" >&2 - : > "$RUN/desired-running.txt" + # API unavailable (featureless or stopped server) — but the supervisor's + # REGISTRY is readable directly, and it is the authority on desired + # state. Read the baseline from sqlite so a desired-running soul with + # no running container is not invisible to every later check: + sqlite3 "$HOME/.freshell/managed-runtime/registry/runtime.sqlite3" \ + "select soul_id from souls where desired_state='running'" \ + > "$RUN/desired-running.txt" 2>/dev/null \ + || { echo "cannot read the registry sqlite — locate desired-running souls manually"; exit 1; } + echo "runtime API unavailable — desired-running baseline read from the registry sqlite ($(wc -l < "$RUN/desired-running.txt") souls)" >&2 fi ``` @@ -664,10 +671,11 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -# The registry stores one row per incarnation, but the API collapses to -# the LATEST row per soul — so normally one row per soul. The grouping -# below stays robust either way: exactly one SOUL must match the probe -# workspace; refuse on ambiguity and act on the last matching row: +# The registry stores one row per incarnation (the API collapses to the +# latest row per soul in the current server). The grouping below is +# deliberately robust under EITHER shape: exactly one SOUL must match +# the probe workspace; refuse on ambiguity and act on the last matching +# row: sids={r["soulId"] for r in rows} assert len(sids)==1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) cur=rows[-1] @@ -766,9 +774,9 @@ PY | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -# The API collapses per-incarnation rows to the latest per soul; the -# grouping stays robust either way — exactly one SOUL per probe -# workspace, refuse on ambiguity, act on the last matching row: +# The grouping is deliberately robust whether the API returns collapsed +# (latest-per-soul) rows or full per-incarnation rows: exactly one SOUL +# per probe workspace, refuse on ambiguity, act on the last matching row: sids={r["soulId"] for r in rows} assert len(sids)==1, ("workspace ambiguous — do not stop anything; resolve manually", rows) s=rows[-1] @@ -802,9 +810,9 @@ print("stopped:", o["outcome"])' | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -# The API collapses per-incarnation rows to the latest per soul; the -# grouping stays robust either way — exactly one SOUL per probe -# workspace, act on the last matching row: +# The grouping is deliberately robust whether the API returns collapsed +# (latest-per-soul) rows or full per-incarnation rows: exactly one SOUL +# per probe workspace, act on the last matching row: sids={r["soulId"] for r in rows} assert len(sids)<=1, ("workspace ambiguous — do not stop anything; resolve manually", rows) if rows: @@ -812,14 +820,18 @@ if rows: print(s["soulId"], s["intentRevision"], s["launchState"], s.get("containerId") or "-") else: print("")' "$BWS") - if [ -n "$BS" ]; then - read -r BS_ID BS_REV BS_STATE BS_CID <<< "$BS" - if [ "$BS_CID" = "-" ]; then - # A probe that failed before container creation owns no - # container — there is nothing to stop; the supervisor has - # no live handle for it: - echo "browser probe soul has no container — nothing to stop" - elif [ "$BS_STATE" != "stopped" ]; then + if [ -z "$BS" ]; then + # Step 7 VERIFIED a running soul in this workspace; a row that + # vanished since is an anomaly, not success — do not delete the + # workspace while the soul's intent is unresolved: + { echo "browser-check soul disappeared from the registry before cleanup — investigate before deleting $BWS"; exit 1; } + fi + read -r BS_ID BS_REV BS_STATE BS_CID <<< "$BS" + if [ "$BS_CID" = "-" ]; then + # No container handle (e.g. a prepared replacement incarnation): + # unresolved intent — refuse rather than report success: + { echo "browser-check soul has no container but state=$BS_STATE — resolve manually before deleting $BWS"; exit 1; } + elif [ "$BS_STATE" != "stopped" ]; then curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$BS_REV}" \ http://127.0.0.1:3001/api/runtime/souls/$BS_ID/stop \ From 810ce582ff2df3f65e8323b53174486524234707 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 14:06:10 -0700 Subject: [PATCH 34/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2032?= =?UTF-8?q?=20=E2=80=94=20cap=20counts=20unique=20soul=20IDs=20(per-incarn?= =?UTF-8?q?ation=20rows=20deduped);=20OpenCode=20default=20cwd=20recorded?= =?UTF-8?q?=20before=20the=20picker=20overwrites=20it=20and=20restored=20i?= =?UTF-8?q?n=20cleanup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 28 +++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index ac69abc1f..94a9ca9e9 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -244,7 +244,10 @@ the procedure mid-way and leave the service down. Two sequences: case "$C" in ''|*[!0-9]*) C=4;; esac # absent/garbage -> supervisor default [ "$C" -ge 1 ] 2>/dev/null || C=1 # supervisor clamps to 1..16 — mirror it [ "$C" -le 16 ] 2>/dev/null || C=16 - N=$(wc -l < "$RUN/desired-running.txt" 2>/dev/null || echo 0) + # Count UNIQUE soul IDs: the inventory stores per-incarnation rows, so + # a raw line count can multiply one soul by its history and inflate the + # cap uselessly: + N=$(sort -u "$RUN/desired-running.txt" 2>/dev/null | wc -l || echo 0) [ "$N" -gt 0 ] || N=$(wc -l < "$RUN/souls-before.txt" 2>/dev/null || echo 0) WC=$(( N * 900 / C * 2 )) [ "$WC" -ge 3600 ] || WC=3600 @@ -467,7 +470,7 @@ for s in souls: souls=json.load(sys.stdin)["souls"] for s in souls: if s.get("desiredState")=="running": - print(s["soulId"])' > "$RUN/desired-running-after.txt" + print(s["soulId"])' | sort -u > "$RUN/desired-running-after.txt" # POSITIVE ownership check for EVERY baseline container: its # (soul, container) pair must appear in the deployed registry with a # RUNNING incarnation — a registry swap or restore leaves containers @@ -658,6 +661,15 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T BWS=$(mktemp -d "$PROBE_ROOT/bws.XXXXXX") # unique per run chmod 0770 "$BWS" # same in-soul-shell access requirement as $WS echo "BWS=$BWS" >> "$D/run.env" + # The directory picker PERSISTS the chosen path as OpenCode's default + # cwd (the confirm handler patches settings) — record the current + # default so step 8 can restore it: + curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ + | python3 -c 'import json,sys +d=json.load(sys.stdin) +d=d.get("data",d) +print((d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") or "")' \ + > "$D/old-opencode-cwd.txt" # Hard-refresh the web client, then in the UI: open the pane picker # and choose OPENCODE specifically — the only managed-enabled # coding-CLI provider in the current release (Claude, Codex, and @@ -842,6 +854,18 @@ print("stopped browser soul:", o["outcome"])' fi fi fi + # Restore OpenCode's default cwd that the directory step overwrote + # (only when the browser check ran — $BWS set): + if [ -n "${BWS:-}" ] && [ -f "$D/old-opencode-cwd.txt" ]; then + OLD=$(cat "$D/old-opencode-cwd.txt") + if [ -n "$OLD" ]; then + curl -s -X PATCH -H @"$D/hdr" -H "Content-Type: application/json" \ + -d '{"codingCli":{"providers":{"opencode":{"cwd":"'"$OLD"'"}}}}' \ + http://127.0.0.1:3001/api/settings > /dev/null + else + echo "OpenCode had no default cwd — clear the probe path in Settings → Coding CLI → OpenCode" + fi + fi rm -rf "$D" "${PROBE_ROOT:-}" "$WS" "${BWS:-}" # success only; both # token headers were already scrubbed by the trap ``` From 4e356d15639ad7902ab0063cb43f7c2771624414 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 14:15:16 -0700 Subject: [PATCH 35/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2033?= =?UTF-8?q?=20=E2=80=94=20settings=20restore=20is=20JSON-safe=20(json.dump?= =?UTF-8?q?s,=20null=20clears),=20HTTP-checked,=20and=20verified=20by=20re?= =?UTF-8?q?-reading=20before=20deletion;=20browser=20check=20waits=20bound?= =?UTF-8?q?ed=20for=20the=20async=20managed=20launch?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 60 +++++++++++++------ 1 file changed, 42 insertions(+), 18 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 94a9ca9e9..0f7b948c5 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -678,21 +678,33 @@ print((d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd # correctly deployed client). A plain shell pane will NOT do either: # shell panes skip the directory step and always take the default # workspace. OpenCode's directory step has a free-text PATH INPUT: - # type the absolute $BWS path there and confirm. Then: + # type the absolute $BWS path there and confirm. Then WAIT (bounded): + # the confirmation dispatches a UI update, terminal.create goes out + # from a React effect, and the managed launch is asynchronous — so + # the soul may not exist the instant you confirm. Poll up to ~120s: + for i in $(seq 1 60); do + if curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys +souls=json.load(sys.stdin)["souls"] +rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] +sids={r["soulId"] for r in rows} +assert len(sids)<=1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) +sys.exit(0 if len(sids)==1 and rows[-1].get("launchState")=="running" else 1)' "$BWS"; then + echo "UI pane is managed (after $((i*2))s)" + break + fi + sleep 2 + done + # Final assert with a clear diagnosis when the wait ran out: curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] -# The registry stores one row per incarnation (the API collapses to the -# latest row per soul in the current server). The grouping below is -# deliberately robust under EITHER shape: exactly one SOUL must match -# the probe workspace; refuse on ambiguity and act on the last matching -# row: sids={r["soulId"] for r in rows} -assert len(sids)==1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) -cur=rows[-1] -assert cur.get("launchState")=="running", "soul not running — client took the host-side route?" -print("UI pane is managed:", cur["soulId"])' "$BWS" +assert len(sids)<=1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) +assert rows and rows[-1].get("launchState")=="running", \ + "no running managed soul for the UI pane after ~120s — client stale or not negotiating" +print("UI pane is managed:", rows[-1]["soulId"])' "$BWS" ``` If the assertion fails, the client is stale or not negotiating @@ -855,16 +867,28 @@ print("stopped browser soul:", o["outcome"])' fi fi # Restore OpenCode's default cwd that the directory step overwrote - # (only when the browser check ran — $BWS set): + # (only when the browser check ran — $BWS set). Build the JSON with + # json.dumps (a path containing quotes/backslashes must not corrupt + # the request; a null CLEARS a previously-absent default), CHECK the + # HTTP status, and VERIFY the saved value by re-reading it — curl + # succeeds as a shell command even on a 4xx/5xx: if [ -n "${BWS:-}" ] && [ -f "$D/old-opencode-cwd.txt" ]; then OLD=$(cat "$D/old-opencode-cwd.txt") - if [ -n "$OLD" ]; then - curl -s -X PATCH -H @"$D/hdr" -H "Content-Type: application/json" \ - -d '{"codingCli":{"providers":{"opencode":{"cwd":"'"$OLD"'"}}}}' \ - http://127.0.0.1:3001/api/settings > /dev/null - else - echo "OpenCode had no default cwd — clear the probe path in Settings → Coding CLI → OpenCode" - fi + BODY=$(python3 -c 'import json,sys +old=sys.argv[1] or None +print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":old}}}}))' "$OLD") + CODE=$(curl -s -o "$D/settings-restore.out" -w '%{http_code}' -X PATCH \ + -H @"$D/hdr" -H "Content-Type: application/json" -d "$BODY" \ + http://127.0.0.1:3001/api/settings) + grep -q '^2' <<<"$CODE" \ + || { echo "settings restore failed (HTTP $CODE):"; cat "$D/settings-restore.out"; exit 1; } + curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ + | python3 -c 'import json,sys +d=json.load(sys.stdin); d=d.get("data",d) +cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") +cur="" if cur is None else cur +sys.exit(0 if cur==sys.argv[1] else 1)' "$OLD" \ + || { echo "settings restore did not stick (expected OpenCode cwd '$OLD') — fix Settings → Coding CLI → OpenCode manually BEFORE deleting the probe dirs"; exit 1; } fi rm -rf "$D" "${PROBE_ROOT:-}" "$WS" "${BWS:-}" # success only; both # token headers were already scrubbed by the trap From f21aa6ae9aed1a86806cc9186db9a42a557da003 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 14:24:31 -0700 Subject: [PATCH 36/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2034?= =?UTF-8?q?=20=E2=80=94=20destructive-path=20guard=20validates=20WS/BWS=20?= =?UTF-8?q?against=20the=20probe=20layout=20before=20any=20stop=20or=20rm;?= =?UTF-8?q?=20recovery=20mode=20clears=20a=20BWS-pointing=20OpenCode=20def?= =?UTF-8?q?ault=20(original=20lost=20with=20the=20shell);=20BWS=20stop=20r?= =?UTF-8?q?equires=20desiredState=20stopped=20too;=20curl=20timeouts=20mak?= =?UTF-8?q?e=20the=20browser=20poll=20actually=20bounded?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 48 +++++++++++++++++-- 1 file changed, 44 insertions(+), 4 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 0f7b948c5..09b5ab043 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -683,7 +683,7 @@ print((d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd # from a React effect, and the managed launch is asynchronous — so # the soul may not exist the instant you confirm. Poll up to ~120s: for i in $(seq 1 60); do - if curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + if curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] @@ -695,8 +695,9 @@ sys.exit(0 if len(sids)==1 and rows[-1].get("launchState")=="running" else 1)' " fi sleep 2 done - # Final assert with a clear diagnosis when the wait ran out: - curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + # Final assert with a clear diagnosis when the wait ran out (the + # --max-time flags are what make the poll actually bounded): + curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] @@ -751,6 +752,20 @@ print("UI pane is managed:", rows[-1]["soulId"])' "$BWS" echo "export WS=, and BWS= if applicable, then re-run" exit 1 fi + # DESTRUCTIVE-PATH GUARD: canonicalize and pattern-match BOTH paths + # against this run's probe layout before any stop request or + # deletion. A mistaken export pointing at a real workspace (one + # managed soul in it) would otherwise stop that live session AND + # rm -rf its project files: + for V in WS BWS; do + eval "VAL=\${$V:-}" + [ -n "$VAL" ] || continue + VAL=$(realpath -e "$VAL") || { echo "$V: path does not exist"; exit 1; } + case "$VAL" in + "$HOME"/freshell-probe.*/ws.*|"$HOME"/freshell-probe.*/bws.*) eval "$V=\$VAL" ;; + *) echo "REFUSING: $V=$VAL is not under a freshell-probe run root — resolve manually"; exit 1 ;; + esac + done D=$(mktemp -d) # fresh records dir for the recovery run fi # Self-healing token header: the trap removed it on the previous exit; @@ -851,11 +866,17 @@ else: { echo "browser-check soul disappeared from the registry before cleanup — investigate before deleting $BWS"; exit 1; } fi read -r BS_ID BS_REV BS_STATE BS_CID <<< "$BS" + # An EXITED incarnation can still be desired-running (eligible for + # recovery) — only a committed stop intent makes it safe to delete + # the workspace, so require desiredState=="stopped" too: + BS_DESIRED=$(curl -s --max-time 10 -H @"$D/hdr" \ + "http://127.0.0.1:3001/api/runtime/souls/$BS_ID" \ + | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"].get("desiredState") or "")') if [ "$BS_CID" = "-" ]; then # No container handle (e.g. a prepared replacement incarnation): # unresolved intent — refuse rather than report success: { echo "browser-check soul has no container but state=$BS_STATE — resolve manually before deleting $BWS"; exit 1; } - elif [ "$BS_STATE" != "stopped" ]; then + elif [ "$BS_STATE" != "stopped" ] || [ "$BS_DESIRED" != "stopped" ]; then curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$BS_REV}" \ http://127.0.0.1:3001/api/runtime/souls/$BS_ID/stop \ @@ -889,6 +910,25 @@ cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") cur="" if cur is None else cur sys.exit(0 if cur==sys.argv[1] else 1)' "$OLD" \ || { echo "settings restore did not stick (expected OpenCode cwd '$OLD') — fix Settings → Coding CLI → OpenCode manually BEFORE deleting the probe dirs"; exit 1; } + elif [ -n "${BWS:-}" ]; then + # RECOVERY MODE: the original $D (with the saved pre-check default) + # was lost with the shell, so the original value is unrecoverable. + # If the current default still points at $BWS, clearing it is the + # only safe option — otherwise deleting $BWS would leave OpenCode + # defaulting into a deleted directory: + CUR=$(curl -s --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ + | python3 -c 'import json,sys +d=json.load(sys.stdin); d=d.get("data",d) +cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") +print("" if cur is None else cur)') + if [ "$CUR" = "$BWS" ]; then + BODY=$(python3 -c 'import json;print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":None}}}}))') + CODE=$(curl -s -o "$D/settings-clear.out" -w '%{http_code}' -X PATCH \ + -H @"$D/hdr" -H "Content-Type: application/json" -d "$BODY" \ + http://127.0.0.1:3001/api/settings) + grep -q '^2' <<<"$CODE" || { echo "clearing OpenCode default failed (HTTP $CODE)"; exit 1; } + echo "NOTE: the pre-check OpenCode default was lost with the lost shell — set your preferred default in Settings → Coding CLI → OpenCode" + fi fi rm -rf "$D" "${PROBE_ROOT:-}" "$WS" "${BWS:-}" # success only; both # token headers were already scrubbed by the trap From c34adac9ed153d69db5a49058cf2448a8e900758 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 14:31:23 -0700 Subject: [PATCH 37/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2035?= =?UTF-8?q?=20=E2=80=94=20extra=20fi=20fixed=20(all=20fenced=20blocks=20ba?= =?UTF-8?q?sh=20-n=20clean);=20destructive-path=20guard=20moved=20to=20the?= =?UTF-8?q?=20common=20path=20(validates=20in=20the=20normal=20flow=20too,?= =?UTF-8?q?=20printf=20-v=20not=20eval);=20bounded=20wait=20for=20the=20pi?= =?UTF-8?q?cker's=20unawaited=20settings=20save=20before=20the=20restore?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 50 +++++++++++++++---- 1 file changed, 39 insertions(+), 11 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 09b5ab043..79cc8e000 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -762,7 +762,7 @@ print("UI pane is managed:", rows[-1]["soulId"])' "$BWS" [ -n "$VAL" ] || continue VAL=$(realpath -e "$VAL") || { echo "$V: path does not exist"; exit 1; } case "$VAL" in - "$HOME"/freshell-probe.*/ws.*|"$HOME"/freshell-probe.*/bws.*) eval "$V=\$VAL" ;; + "$HOME"/freshell-probe.*/ws.*|"$HOME"/freshell-probe.*/bws.*) printf -v "$V" '%s' "$VAL" ;; *) echo "REFUSING: $V=$VAL is not under a freshell-probe run root — resolve manually"; exit 1 ;; esac done @@ -798,6 +798,20 @@ PY umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" fi trap 'rm -f "${RUN:-/nonexistent}/hdr" "${D:-/nonexistent}/hdr"' EXIT # BOTH headers, every exit path, set -u-safe + # DESTRUCTIVE-PATH GUARD (both the normal and recovery flows pass + # through here): canonicalize and pattern-match every probe path + # against this run's layout BEFORE any stop request or deletion. A + # mistaken variable pointing at a real workspace (one managed soul in + # it) would otherwise stop that live session AND rm -rf its files: + for V in WS BWS; do + eval "VAL=\${$V:-}" + [ -n "$VAL" ] || continue + VAL=$(realpath -e "$VAL") || { echo "$V: path does not exist"; exit 1; } + case "$VAL" in + "$HOME"/freshell-probe.*/ws.*|"$HOME"/freshell-probe.*/bws.*) printf -v "$V" '%s' "$VAL" ;; + *) echo "REFUSING: $V=$VAL is not under a freshell-probe run root — resolve manually"; exit 1 ;; + esac + done if [ -z "${SOUL_ID:-}" ]; then # Recovery mode: derive the probe soul from the unique workspace — # no independent terminalId check applies here; WS uniqueness IS @@ -877,22 +891,36 @@ else: # unresolved intent — refuse rather than report success: { echo "browser-check soul has no container but state=$BS_STATE — resolve manually before deleting $BWS"; exit 1; } elif [ "$BS_STATE" != "stopped" ] || [ "$BS_DESIRED" != "stopped" ]; then - curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ - -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$BS_REV}" \ - http://127.0.0.1:3001/api/runtime/souls/$BS_ID/stop \ - | python3 -c 'import json,sys + curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ + -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$BS_REV}" \ + http://127.0.0.1:3001/api/runtime/souls/$BS_ID/stop \ + | python3 -c 'import json,sys o=json.load(sys.stdin) assert o.get("outcome")=="verified_empty", o print("stopped browser soul:", o["outcome"])' - fi fi fi # Restore OpenCode's default cwd that the directory step overwrote - # (only when the browser check ran — $BWS set). Build the JSON with - # json.dumps (a path containing quotes/backslashes must not corrupt - # the request; a null CLEARS a previously-absent default), CHECK the - # HTTP status, and VERIFY the saved value by re-reading it — curl - # succeeds as a shell command even on a 4xx/5xx: + # (only when the browser check ran — $BWS set). The picker's settings + # save is an UNAWAITED async request — first WAIT (bounded) for it to + # land (the visible default becomes $BWS), or the delayed save could + # arrive AFTER this restore and re-point the default at the + # about-to-be-deleted workspace: + if [ -n "${BWS:-}" ]; then + for i in $(seq 1 15); do + CUR=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ + | python3 -c 'import json,sys +d=json.load(sys.stdin); d=d.get("data",d) +cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") +print("" if cur is None else cur)') + if [ "$CUR" = "$BWS" ]; then break; fi + sleep 2 + done + fi + # Then the restore itself — JSON-safe (a path containing + # quotes/backslashes must not corrupt the request; a null CLEARS a + # previously-absent default), HTTP-status-checked, and VERIFIED by + # re-reading it — curl succeeds as a shell command even on a 4xx/5xx: if [ -n "${BWS:-}" ] && [ -f "$D/old-opencode-cwd.txt" ]; then OLD=$(cat "$D/old-opencode-cwd.txt") BODY=$(python3 -c 'import json,sys From 6158910e0ec1c52d6242c98cb5c375c94d44bccf Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 14:38:19 -0700 Subject: [PATCH 38/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2036?= =?UTF-8?q?=20=E2=80=94=20PROBE=5FROOT=20pattern-validated=20before=20chmo?= =?UTF-8?q?d=20and=20included=20in=20the=20cleanup=20guard;=20picker-save?= =?UTF-8?q?=20poll=20exhaustion=20aborts=20cleanup=20instead=20of=20racing?= =?UTF-8?q?=20a=20still-in-flight=20save?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 21 +++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 79cc8e000..518e45945 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -657,7 +657,13 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T ```bash : "${PROBE_ROOT:=$(mktemp -d ~/freshell-probe.XXXXXX)}" - chmod 0770 "$PROBE_ROOT" # only ever chmods a directory this run made + # An environment-inherited PROBE_ROOT is only trusted after the same + # pattern check the cleanup guard applies — never chmod (or later + # rm -rf) an unvalidated directory: + case "$PROBE_ROOT" in + "$HOME"/freshell-probe.*) chmod 0770 "$PROBE_ROOT" ;; + *) echo "REFUSING: PROBE_ROOT=$PROBE_ROOT is not a freshell-probe run root"; exit 1 ;; + esac BWS=$(mktemp -d "$PROBE_ROOT/bws.XXXXXX") # unique per run chmod 0770 "$BWS" # same in-soul-shell access requirement as $WS echo "BWS=$BWS" >> "$D/run.env" @@ -762,7 +768,7 @@ print("UI pane is managed:", rows[-1]["soulId"])' "$BWS" [ -n "$VAL" ] || continue VAL=$(realpath -e "$VAL") || { echo "$V: path does not exist"; exit 1; } case "$VAL" in - "$HOME"/freshell-probe.*/ws.*|"$HOME"/freshell-probe.*/bws.*) printf -v "$V" '%s' "$VAL" ;; + "$HOME"/freshell-probe.*) printf -v "$V" '%s' "$VAL" ;; *) echo "REFUSING: $V=$VAL is not under a freshell-probe run root — resolve manually"; exit 1 ;; esac done @@ -803,12 +809,12 @@ PY # against this run's layout BEFORE any stop request or deletion. A # mistaken variable pointing at a real workspace (one managed soul in # it) would otherwise stop that live session AND rm -rf its files: - for V in WS BWS; do + for V in WS BWS PROBE_ROOT; do eval "VAL=\${$V:-}" [ -n "$VAL" ] || continue VAL=$(realpath -e "$VAL") || { echo "$V: path does not exist"; exit 1; } case "$VAL" in - "$HOME"/freshell-probe.*/ws.*|"$HOME"/freshell-probe.*/bws.*) printf -v "$V" '%s' "$VAL" ;; + "$HOME"/freshell-probe.*) printf -v "$V" '%s' "$VAL" ;; *) echo "REFUSING: $V=$VAL is not under a freshell-probe run root — resolve manually"; exit 1 ;; esac done @@ -916,6 +922,13 @@ print("" if cur is None else cur)') if [ "$CUR" = "$BWS" ]; then break; fi sleep 2 done + if [ "$CUR" != "$BWS" ]; then + # The save state is unknown — a still-in-flight picker save could + # re-point the default AFTER our restore, into a deleted dir. Do + # not proceed; wait for the save to settle and re-run cleanup: + echo "OpenCode default never settled on the probe path — the picker's async save may still be in flight; wait, then re-run this step" + exit 1 + fi fi # Then the restore itself — JSON-safe (a path containing # quotes/backslashes must not corrupt the request; a null CLEARS a From c6604856969ff4eb11a18def7c94f60012635224 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 14:46:53 -0700 Subject: [PATCH 39/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2037?= =?UTF-8?q?=20=E2=80=94=20guard=20ties=20WS/BWS/PROBE=5FROOT=20to=20one=20?= =?UTF-8?q?canonical=20probe=20root=20(no=20cross-run=20deletion);=20PROBE?= =?UTF-8?q?=5FROOT=20realpath'd=20before=20chmod=20(symlinks);=20reset-fai?= =?UTF-8?q?led=20for=20the=20start-rate-limit=20state;=20headers=20scrubbe?= =?UTF-8?q?d=20explicitly=20on=20success?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 47 +++++++++++++------ 1 file changed, 32 insertions(+), 15 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 518e45945..a9e329bbe 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -323,9 +323,13 @@ drop-in, plus `freshell-supervisor.service`): `systemctl --user restart freshell-supervisor.service` redeploys both. The same `Type=simple` socket race as the strict sequence applies here: the server may start before the supervisor rebinds its control socket - and come up failed. If the server unit is not `active` after the - coupled bounce, connect-probe the socket as in the strict sequence and - `systemctl --user start freshell-rust.service` once it is live. + and come up failed; the unit's `Restart=on-failure` retries can then + trip systemd's start-rate limit, after which a plain `start` is + REFUSED. If the server unit is not `active` after the coupled bounce: + connect-probe the socket as in the strict sequence, then + `systemctl --user reset-failed freshell-rust.service` (clears the + rate-limit counter) and `systemctl --user start freshell-rust.service` + once it is live. - Restarting the server alone (`systemctl --user restart freshell-rust.service`) leaves the supervisor running. Use this for server-only changes. @@ -657,9 +661,11 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T ```bash : "${PROBE_ROOT:=$(mktemp -d ~/freshell-probe.XXXXXX)}" - # An environment-inherited PROBE_ROOT is only trusted after the same - # pattern check the cleanup guard applies — never chmod (or later - # rm -rf) an unvalidated directory: + # An environment-inherited PROBE_ROOT is only trusted after + # canonicalization (defeat symlinks — chmod follows them) and the + # pattern check: + PROBE_ROOT=$(realpath -e "$PROBE_ROOT") \ + || { echo "PROBE_ROOT: path does not exist"; exit 1; } case "$PROBE_ROOT" in "$HOME"/freshell-probe.*) chmod 0770 "$PROBE_ROOT" ;; *) echo "REFUSING: PROBE_ROOT=$PROBE_ROOT is not a freshell-probe run root"; exit 1 ;; @@ -805,18 +811,27 @@ PY fi trap 'rm -f "${RUN:-/nonexistent}/hdr" "${D:-/nonexistent}/hdr"' EXIT # BOTH headers, every exit path, set -u-safe # DESTRUCTIVE-PATH GUARD (both the normal and recovery flows pass - # through here): canonicalize and pattern-match every probe path - # against this run's layout BEFORE any stop request or deletion. A - # mistaken variable pointing at a real workspace (one managed soul in - # it) would otherwise stop that live session AND rm -rf its files: + # through here): canonicalize every probe path, pattern-match it + # against the freshell-probe layout, AND tie WS/BWS/PROBE_ROOT to ONE + # shared root — validating them independently would let a mixed-up + # recovery export stop another run's soul and delete its directories: + GUARD_ROOT="" for V in WS BWS PROBE_ROOT; do eval "VAL=\${$V:-}" [ -n "$VAL" ] || continue VAL=$(realpath -e "$VAL") || { echo "$V: path does not exist"; exit 1; } - case "$VAL" in - "$HOME"/freshell-probe.*) printf -v "$V" '%s' "$VAL" ;; - *) echo "REFUSING: $V=$VAL is not under a freshell-probe run root — resolve manually"; exit 1 ;; + PARENT=$(dirname "$VAL") + case "$PARENT" in + "$HOME"/freshell-probe.*) ;; + *) echo "REFUSING: $V=$VAL is not a freshell-probe workspace/root — resolve manually"; exit 1 ;; esac + if [ -z "$GUARD_ROOT" ]; then + GUARD_ROOT="$PARENT" + elif [ "$PARENT" != "$GUARD_ROOT" ]; then + echo "REFUSING: $V=$VAL belongs to a different probe root than the earlier paths — resolve manually" + exit 1 + fi + printf -v "$V" '%s' "$VAL" done if [ -z "${SOUL_ID:-}" ]; then # Recovery mode: derive the probe soul from the unique workspace — @@ -971,8 +986,10 @@ print("" if cur is None else cur)') echo "NOTE: the pre-check OpenCode default was lost with the lost shell — set your preferred default in Settings → Coding CLI → OpenCode" fi fi - rm -rf "$D" "${PROBE_ROOT:-}" "$WS" "${BWS:-}" # success only; both - # token headers were already scrubbed by the trap + rm -f "${RUN:-/nonexistent}/hdr" "$D/hdr" # scrub NOW; the trap only + # fires at shell EXIT, which is too late for $RUN/hdr in a still- + # living interactive shell + rm -rf "$D" "${PROBE_ROOT:-}" "$WS" "${BWS:-}" # success only ``` If the block aborts before `verified_empty`, the token header is From 103cf60ee4b8c0d7e4ea402fe0f0f238bbe50197 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 14:52:49 -0700 Subject: [PATCH 40/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2038?= =?UTF-8?q?=20=E2=80=94=20guard=20fixed=20for=20the=20normal=20flow=20(PRO?= =?UTF-8?q?BE=5FROOT=20IS=20the=20root;=20WS/BWS=20must=20sit=20under=20it?= =?UTF-8?q?)=20and=20now=20covers=20D=20(mktemp=20+=20probe=20records=20re?= =?UTF-8?q?quired);=20guard=20logic=20executed=20against=20fixture=20paths?= =?UTF-8?q?,=20all=20seven=20cases=20behave?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 33 +++++++++++++++---- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index a9e329bbe..badda9662 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -811,28 +811,47 @@ PY fi trap 'rm -f "${RUN:-/nonexistent}/hdr" "${D:-/nonexistent}/hdr"' EXIT # BOTH headers, every exit path, set -u-safe # DESTRUCTIVE-PATH GUARD (both the normal and recovery flows pass - # through here): canonicalize every probe path, pattern-match it - # against the freshell-probe layout, AND tie WS/BWS/PROBE_ROOT to ONE - # shared root — validating them independently would let a mixed-up - # recovery export stop another run's soul and delete its directories: + # through here). Canonicalize every probe path and tie them together: + # PROBE_ROOT must BE a freshell-probe run root; WS and BWS must be + # workspaces DIRECTLY UNDER that same root; D must be a mktemp-style + # records directory carrying this run's probe records. Validating the + # paths independently would let a mixed-up recovery export stop + # another run's soul and delete its files: GUARD_ROOT="" - for V in WS BWS PROBE_ROOT; do + if [ -n "${PROBE_ROOT:-}" ]; then + PROBE_ROOT=$(realpath -e "$PROBE_ROOT") || { echo "PROBE_ROOT: path does not exist"; exit 1; } + case "$PROBE_ROOT" in + "$HOME"/freshell-probe.*) GUARD_ROOT="$PROBE_ROOT" ;; + *) echo "REFUSING: PROBE_ROOT=$PROBE_ROOT is not a freshell-probe run root"; exit 1 ;; + esac + fi + for V in WS BWS; do eval "VAL=\${$V:-}" [ -n "$VAL" ] || continue VAL=$(realpath -e "$VAL") || { echo "$V: path does not exist"; exit 1; } PARENT=$(dirname "$VAL") case "$PARENT" in "$HOME"/freshell-probe.*) ;; - *) echo "REFUSING: $V=$VAL is not a freshell-probe workspace/root — resolve manually"; exit 1 ;; + *) echo "REFUSING: $V=$VAL is not a freshell-probe workspace — resolve manually"; exit 1 ;; esac if [ -z "$GUARD_ROOT" ]; then GUARD_ROOT="$PARENT" elif [ "$PARENT" != "$GUARD_ROOT" ]; then - echo "REFUSING: $V=$VAL belongs to a different probe root than the earlier paths — resolve manually" + echo "REFUSING: $V=$VAL belongs to a different probe root than the other paths — resolve manually" exit 1 fi printf -v "$V" '%s' "$VAL" done + if [ -n "${D:-}" ]; then + case "$D" in + /tmp/tmp.*) ;; + *) echo "REFUSING: D=$D is not a mktemp records directory"; exit 1 ;; + esac + if [ ! -f "$D/probe-soul-id.txt" ] && [ ! -f "$D/run.env" ]; then + echo "REFUSING: D=$D carries no probe records — resolve manually" + exit 1 + fi + fi if [ -z "${SOUL_ID:-}" ]; then # Recovery mode: derive the probe soul from the unique workspace — # no independent terminalId check applies here; WS uniqueness IS From 9a8e34cd686625136826d6b1a8f82f252ba0d989 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 15:02:36 -0700 Subject: [PATCH 41/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2039?= =?UTF-8?q?=20=E2=80=94=20recovery-minted=20D=20exempt=20from=20the=20reco?= =?UTF-8?q?rds=20check=20(recovery=20runs=20again);=20D's=20run.env=20work?= =?UTF-8?q?space=20cross-checked=20against=20WS=20(mixed-run=20D+WS=20refu?= =?UTF-8?q?sed);=20fixtures=20extended,=20all=20pass?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index badda9662..29d37bd8a 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -779,6 +779,7 @@ print("UI pane is managed:", rows[-1]["soulId"])' "$BWS" esac done D=$(mktemp -d) # fresh records dir for the recovery run + RECOVERY_MINTED=1 # exempt from the records check (it is empty BY DESIGN) fi # Self-healing token header: the trap removed it on the previous exit; # recreate it (0600) only when missing, extracting the token exactly @@ -847,9 +848,21 @@ PY /tmp/tmp.*) ;; *) echo "REFUSING: D=$D is not a mktemp records directory"; exit 1 ;; esac - if [ ! -f "$D/probe-soul-id.txt" ] && [ ! -f "$D/run.env" ]; then - echo "REFUSING: D=$D carries no probe records — resolve manually" - exit 1 + if [ "${RECOVERY_MINTED:-0}" != "1" ]; then + if [ ! -f "$D/probe-soul-id.txt" ] && [ ! -f "$D/run.env" ]; then + echo "REFUSING: D=$D carries no probe records — resolve manually" + exit 1 + fi + # TIE the records to THIS run's workspace: mixing one run's D (and + # its self-consistent SOUL_ID/TID) with another run's WS would + # otherwise stop the old run's soul and delete the new workspace: + if [ -n "${WS:-}" ] && [ -f "$D/run.env" ]; then + RECORDED_WS=$(sed -n 's/^WS=//p' "$D/run.env" | tail -1) + if [ -n "$RECORDED_WS" ] && [ "$RECORDED_WS" != "$WS" ]; then + echo "REFUSING: D=$D records workspace $RECORDED_WS, but WS=$WS — mixed runs; resolve manually" + exit 1 + fi + fi fi fi if [ -z "${SOUL_ID:-}" ]; then From a19e0fb23caee33d82eedba1b91159b178c7392c Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 15:13:50 -0700 Subject: [PATCH 42/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2040?= =?UTF-8?q?=20=E2=80=94=20probe=20workspaces=20must=20be=20free=20of=20vis?= =?UTF-8?q?ible=20user=20files=20(a=20name=20pattern=20is=20not=20ownershi?= =?UTF-8?q?p);=20recovery=20confirms=20the=20matched=20soul=20record=20bef?= =?UTF-8?q?ore=20the=20stop;=20browser=20poll=20on=20an=20elapsed-time=201?= =?UTF-8?q?20s=20deadline;=20every=20curl=20carries=20connect=20and=20tota?= =?UTF-8?q?l=20timeouts?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 55 ++++++++++++------- 1 file changed, 34 insertions(+), 21 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 29d37bd8a..02cd5d1cc 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -165,9 +165,9 @@ else : > "$RUN/desired-running.txt" fi trap 'rm -f "${RUN:-/nonexistent}/hdr"' EXIT # ${RUN:-} survives set -u -if [ -f "$RUN/hdr" ] && curl -s -o /dev/null -w '%{http_code}' -H @"$RUN/hdr" \ +if [ -f "$RUN/hdr" ] && curl -s --connect-timeout 5 --max-time 10 -o /dev/null -w '%{http_code}' -H @"$RUN/hdr" \ http://127.0.0.1:3001/api/runtime/readiness | grep -q 200; then - curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ + curl -s --connect-timeout 5 --max-time 10 -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] for s in souls: @@ -464,12 +464,12 @@ PY # from the registry is exactly what the sweep must catch (this also # fills the API-less fallback's empty baseline — an empty sweep # would otherwise pass vacuously): - curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ + curl -s --connect-timeout 5 --max-time 10 -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] for s in souls: print(s["soulId"], s.get("containerId") or "-", s.get("launchState") or "-")' > "$RUN/souls-all.txt" - curl -s -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ + curl -s --connect-timeout 5 --max-time 10 -H @"$RUN/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] for s in souls: @@ -492,7 +492,7 @@ for s in souls: sort -u "$RUN/desired-running.txt" "$RUN/desired-running-after.txt" \ | while read -r SID; do [ -n "$SID" ] || continue - curl -s -H @"$RUN/hdr" "http://127.0.0.1:3001/api/runtime/souls/$SID" \ + curl -s --connect-timeout 5 --max-time 10 -H @"$RUN/hdr" "http://127.0.0.1:3001/api/runtime/souls/$SID" \ | python3 -c 'import json,sys o=json.load(sys.stdin) s=o.get("soul") @@ -580,11 +580,11 @@ PY trap 'rm -f "${RUN:-/nonexistent}/hdr" "${D:-/nonexistent}/hdr"' EXIT printf 'D=%q\nWS=%q\nCHECKOUT=%q\n' "$D" "$WS" "$CHECKOUT" > "$D/run.env" # create the probe pane (example); record terminalId from the response: - curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ + curl -s --connect-timeout 5 --max-time 30 -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d '{"mode":"shell","cwd":"'"$WS"'","name":"home-mount probe"}' \ http://127.0.0.1:3001/api/tabs | tee "$D/probe-tab.json" TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") - SOUL_ID=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + SOUL_ID=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$TID" \ @@ -676,7 +676,7 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T # The directory picker PERSISTS the chosen path as OpenCode's default # cwd (the confirm handler patches settings) — record the current # default so step 8 can restore it: - curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ + curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ | python3 -c 'import json,sys d=json.load(sys.stdin) d=d.get("data",d) @@ -694,7 +694,8 @@ print((d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd # the confirmation dispatches a UI update, terminal.create goes out # from a React effect, and the managed launch is asynchronous — so # the soul may not exist the instant you confirm. Poll up to ~120s: - for i in $(seq 1 60); do + POLL_DEADLINE=$((SECONDS + 120)) + while [ "$SECONDS" -lt "$POLL_DEADLINE" ]; do if curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] @@ -702,7 +703,7 @@ rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] sids={r["soulId"] for r in rows} assert len(sids)<=1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) sys.exit(0 if len(sids)==1 and rows[-1].get("launchState")=="running" else 1)' "$BWS"; then - echo "UI pane is managed (after $((i*2))s)" + echo "UI pane is managed (after $((SECONDS - POLL_DEADLINE + 120))s)" break fi sleep 2 @@ -835,6 +836,13 @@ PY "$HOME"/freshell-probe.*) ;; *) echo "REFUSING: $V=$VAL is not a freshell-probe workspace — resolve manually"; exit 1 ;; esac + # A NAME pattern is not ownership: a real workspace could sit under + # a freshell-probe-named directory. Probe workspaces contain NO + # visible (non-dotfile) content — refuse anything with user files: + if [ -n "$(find "$VAL" -mindepth 1 -maxdepth 2 -not -name '.*' -print -quit 2>/dev/null)" ]; then + echo "REFUSING: $V=$VAL contains visible files — probe workspaces are empty; resolve manually" + exit 1 + fi if [ -z "$GUARD_ROOT" ]; then GUARD_ROOT="$PARENT" elif [ "$PARENT" != "$GUARD_ROOT" ]; then @@ -876,7 +884,7 @@ PY echo "export WS=, and BWS= if applicable, then re-run" exit 1 fi - PROBE=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + PROBE=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] @@ -888,19 +896,24 @@ assert len(sids)==1, ("workspace ambiguous — do not stop anything; resolve man s=rows[-1] print(s["soulId"], s["intentRevision"], s["terminalId"])' "$WS") read -r SOUL_ID REV TID <<< "$PROBE" + # CONFIRM the matched record is really the throwaway probe — the + # final human check before a destructive stop: + echo "Recovery matched: soul=$SOUL_ID terminal=$TID workspace=$WS" + read -r -p "Stop and delete this PROBE soul? Type 'yes' to confirm: " CONFIRM + [ "$CONFIRM" = "yes" ] || { echo "aborted — nothing stopped, nothing deleted"; exit 1; } else TID=$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1]))["data"]["terminalId"])' "$D/probe-tab.json") # Identity re-check FIRST; with set -e its failure aborts the block # before the revision fetch and the stop request are ever reached: - curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ + curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ | python3 -c 'import json,sys s=json.load(sys.stdin)["soul"] assert s.get("terminalId")==sys.argv[1], ("SOUL/PROBE MISMATCH", s) print("identity ok:", s.get("terminalId"))' "$TID" - REV=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ + REV=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID \ | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"]["intentRevision"])') fi - curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ + curl -s --connect-timeout 5 --max-time 30 -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$REV}" \ http://127.0.0.1:3001/api/runtime/souls/$SOUL_ID/stop \ | python3 -c 'import json,sys @@ -912,7 +925,7 @@ print("stopped:", o["outcome"])' # Look it up by workspace in ANY launchState — a blocked or exited # probe soul still must not be orphaned against a deleted workspace: if [ -n "${BWS:-}" ]; then - BS=$(curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ + BS=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] @@ -936,7 +949,7 @@ else: # An EXITED incarnation can still be desired-running (eligible for # recovery) — only a committed stop intent makes it safe to delete # the workspace, so require desiredState=="stopped" too: - BS_DESIRED=$(curl -s --max-time 10 -H @"$D/hdr" \ + BS_DESIRED=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" \ "http://127.0.0.1:3001/api/runtime/souls/$BS_ID" \ | python3 -c 'import json,sys;print(json.load(sys.stdin)["soul"].get("desiredState") or "")') if [ "$BS_CID" = "-" ]; then @@ -944,7 +957,7 @@ else: # unresolved intent — refuse rather than report success: { echo "browser-check soul has no container but state=$BS_STATE — resolve manually before deleting $BWS"; exit 1; } elif [ "$BS_STATE" != "stopped" ] || [ "$BS_DESIRED" != "stopped" ]; then - curl -s -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ + curl -s --connect-timeout 5 --max-time 30 -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d "{\"requestId\":\"probe-cleanup\",\"expectedIntentRevision\":$BS_REV}" \ http://127.0.0.1:3001/api/runtime/souls/$BS_ID/stop \ | python3 -c 'import json,sys @@ -986,12 +999,12 @@ print("" if cur is None else cur)') BODY=$(python3 -c 'import json,sys old=sys.argv[1] or None print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":old}}}}))' "$OLD") - CODE=$(curl -s -o "$D/settings-restore.out" -w '%{http_code}' -X PATCH \ + CODE=$(curl -s --connect-timeout 5 --max-time 30 -o "$D/settings-restore.out" -w '%{http_code}' -X PATCH \ -H @"$D/hdr" -H "Content-Type: application/json" -d "$BODY" \ http://127.0.0.1:3001/api/settings) grep -q '^2' <<<"$CODE" \ || { echo "settings restore failed (HTTP $CODE):"; cat "$D/settings-restore.out"; exit 1; } - curl -s -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ + curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ | python3 -c 'import json,sys d=json.load(sys.stdin); d=d.get("data",d) cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") @@ -1004,14 +1017,14 @@ sys.exit(0 if cur==sys.argv[1] else 1)' "$OLD" \ # If the current default still points at $BWS, clearing it is the # only safe option — otherwise deleting $BWS would leave OpenCode # defaulting into a deleted directory: - CUR=$(curl -s --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ + CUR=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ | python3 -c 'import json,sys d=json.load(sys.stdin); d=d.get("data",d) cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") print("" if cur is None else cur)') if [ "$CUR" = "$BWS" ]; then BODY=$(python3 -c 'import json;print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":None}}}}))') - CODE=$(curl -s -o "$D/settings-clear.out" -w '%{http_code}' -X PATCH \ + CODE=$(curl -s --connect-timeout 5 --max-time 30 -o "$D/settings-clear.out" -w '%{http_code}' -X PATCH \ -H @"$D/hdr" -H "Content-Type: application/json" -d "$BODY" \ http://127.0.0.1:3001/api/settings) grep -q '^2' <<<"$CODE" || { echo "clearing OpenCode default failed (HTTP $CODE)"; exit 1; } From 4efd6213d417e8ca2fe6925a2be7af9186ebfc60 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 15:24:01 -0700 Subject: [PATCH 43/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2041?= =?UTF-8?q?=20=E2=80=94=20SAFE-11=20codex=20retention=20truth=20(systemd?= =?UTF-8?q?=20cgroup=20kill=20defeats=20it;=20unit=20KillMode=20caveat);?= =?UTF-8?q?=20probe=20root=20deleted=20only=20when=20it=20holds=20nothing?= =?UTF-8?q?=20but=20this=20run's=20workspaces;=20poll=20clock=20starts=20a?= =?UTF-8?q?fter=20the=20operator's=20UI=20step;=20strict=20build=20ordered?= =?UTF-8?q?=20after=20the=20stop;=20health=20curls=20bounded?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 37 ++++++++++++++++--- 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 02cd5d1cc..45fb57d51 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -32,7 +32,7 @@ Run this on the machine that owns the server: ```bash cd /home/dan/code/freshell -curl -fsS http://127.0.0.1:3001/api/health +curl -fsS --connect-timeout 5 --max-time 10 http://127.0.0.1:3001/api/health ``` If the health check fails, start the server with the canonical launcher: @@ -201,9 +201,12 @@ the procedure mid-way and leave the service down. Two sequences: - Quick (compatible changes): build, then restart immediately, accepting the short skew window. -- Strict (changes that must never skew): `systemctl --user stop - freshell-supervisor.service` (the server stops with it via the `Requires=` - coupling), build, then start in socket order: start BOTH units — +- Strict (changes that must never skew): FIRST `systemctl --user stop + freshell-supervisor.service` (the server stops with it via the + `Requires=` coupling), THEN run the build command from the Build + surface above — never before the stop, or the running supervisor + picks up the new session host mid-flight (the exact skew this + sequence exists to prevent) — then start in socket order: start BOTH units — `Requires=` pulls the supervisor in when the server starts, but starting the supervisor alone does not start its dependent server, so starting only the supervisor would leave the web server down. The supervisor @@ -380,7 +383,7 @@ drop-in, plus `freshell-supervisor.service`): **Post-deploy verification checklist:** -1. Health: `curl -fsS http://127.0.0.1:3001/api/health` → `"ready": true`. +1. Health: `curl -fsS --connect-timeout 5 --max-time 10 http://127.0.0.1:3001/api/health` → `"ready": true`. 2. Feature present AND scan completed: with the auth token, `GET /api/runtime/readiness` answers `200` — a `404` means the running binary was built WITHOUT the feature; rebuild with the command above @@ -694,6 +697,9 @@ print((d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd # the confirmation dispatches a UI update, terminal.create goes out # from a React effect, and the managed launch is asynchronous — so # the soul may not exist the instant you confirm. Poll up to ~120s: + # The manual UI step must finish BEFORE the clock starts — create + # the pane, then press ENTER to start the bounded poll: + read -r -p "Create the OpenCode pane at $BWS in the hard-refreshed client, then press ENTER to poll: " _ POLL_DEADLINE=$((SECONDS + 120)) while [ "$SECONDS" -lt "$POLL_DEADLINE" ]; do if curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ @@ -1034,7 +1040,26 @@ print("" if cur is None else cur)') rm -f "${RUN:-/nonexistent}/hdr" "$D/hdr" # scrub NOW; the trap only # fires at shell EXIT, which is too late for $RUN/hdr in a still- # living interactive shell - rm -rf "$D" "${PROBE_ROOT:-}" "$WS" "${BWS:-}" # success only + # Delete the probe workspaces, and the ROOT only when it holds + # nothing besides this run's known workspaces — a root with other + # entries (another workspace, hidden data) is NOT ours to delete: + rm -rf "$D" "$WS" "${BWS:-}" + if [ -n "${PROBE_ROOT:-}" ]; then + UNEXPECTED="" + shopt -s nullglob + for ENTRY in "$PROBE_ROOT"/* "$PROBE_ROOT"/.[!.]*; do + case "$ENTRY" in + "$WS"|"$BWS") ;; + *) UNEXPECTED="$UNEXPECTED $ENTRY" ;; + esac + done + shopt -u nullglob + if [ -z "$UNEXPECTED" ]; then + rm -rf "$PROBE_ROOT" + else + echo "probe root kept — it contains unexpected entries:$UNEXPECTED" + fi + fi ``` If the block aborts before `verified_empty`, the token header is From d5c131524b78419c1252a40effdd7c74b807ea44 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 15:32:49 -0700 Subject: [PATCH 44/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2042?= =?UTF-8?q?=20=E2=80=94=20probe=20workspaces=20must=20be=20completely=20em?= =?UTF-8?q?pty=20(dotfiles=20included;=20in-soul=20state=20lives=20in=20th?= =?UTF-8?q?e=20soul=20volume);=20root-entry=20sweep=20covers=20..-prefixed?= =?UTF-8?q?=20names;=20codex=20retention=20warning=20placed=20at=20the=20r?= =?UTF-8?q?estart=20instructions=20as=20well?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/development/self-hosted-launch-runbook.md | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 45fb57d51..baa200d23 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -324,6 +324,11 @@ drop-in, plus `freshell-supervisor.service`): `freshell-supervisor.service`. Restarting the SUPERVISOR therefore also restarts the server in the same transaction — one `systemctl --user restart freshell-supervisor.service` redeploys both. + BEWARE: any server-unit restart terminates the server's RETAINED Codex + sidecars (see the restart-effects list below — the unit has no + `KillMode` override, so systemd's control-group kill defeats the + server's deliberate SAFE-11 retention); in-flight Codex conversations + are lost. The same `Type=simple` socket race as the strict sequence applies here: the server may start before the supervisor rebinds its control socket and come up failed; the unit's `Restart=on-failure` retries can then @@ -843,10 +848,13 @@ PY *) echo "REFUSING: $V=$VAL is not a freshell-probe workspace — resolve manually"; exit 1 ;; esac # A NAME pattern is not ownership: a real workspace could sit under - # a freshell-probe-named directory. Probe workspaces contain NO - # visible (non-dotfile) content — refuse anything with user files: - if [ -n "$(find "$VAL" -mindepth 1 -maxdepth 2 -not -name '.*' -print -quit 2>/dev/null)" ]; then - echo "REFUSING: $V=$VAL contains visible files — probe workspaces are empty; resolve manually" + # a freshell-probe-named directory. Probe workspaces are created + # EMPTY and stay empty (in-soul provider state lives in the soul + # volume, not the workspace) — refuse ANY content, dotfiles + # included, so a mistaken selection holding e.g. only an .env is + # still caught: + if [ -n "$(find "$VAL" -mindepth 1 -print -quit 2>/dev/null)" ]; then + echo "REFUSING: $V=$VAL is not empty — probe workspaces are empty; resolve manually" exit 1 fi if [ -z "$GUARD_ROOT" ]; then @@ -1047,7 +1055,7 @@ print("" if cur is None else cur)') if [ -n "${PROBE_ROOT:-}" ]; then UNEXPECTED="" shopt -s nullglob - for ENTRY in "$PROBE_ROOT"/* "$PROBE_ROOT"/.[!.]*; do + for ENTRY in "$PROBE_ROOT"/* "$PROBE_ROOT"/.[!.]* "$PROBE_ROOT"/..?*; do case "$ENTRY" in "$WS"|"$BWS") ;; *) UNEXPECTED="$UNEXPECTED $ENTRY" ;; From 145f717a43bb31e02dd3c28ff2867f3312ff898c Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 15:41:53 -0700 Subject: [PATCH 45/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2043?= =?UTF-8?q?=20=E2=80=94=20per-unit=20is-active=20checks=20(multi-unit=20fo?= =?UTF-8?q?rm=20passes=20when=20any=20one=20is=20up);=20D=20pattern=20hono?= =?UTF-8?q?rs=20TMPDIR;=20run.env=20WS=20read=20by=20sourcing=20(unescapes?= =?UTF-8?q?=20%q,=20spaced=20paths=20tie=20correctly);=20fixtures=20cover?= =?UTF-8?q?=20both?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/development/self-hosted-launch-runbook.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index baa200d23..95aad1242 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -398,8 +398,12 @@ drop-in, plus `freshell-supervisor.service`): completed without subsystem errors, NOT that every soul can reattach — an individually blocked soul is a per-soul outcome that does not enter `blockedSubsystems`, which is why step 4 checks each soul directly. -3. Units: `systemctl --user is-active freshell-rust.service - freshell-supervisor.service` → both `active`, and the supervisor's +3. Units: `is-active` with MULTIPLE units succeeds if at least ONE is + active, so check each separately and assert both outputs are + `active`: + `systemctl --user is-active freshell-rust.service` and + `systemctl --user is-active freshell-supervisor.service` → each + `active`, and the supervisor's control socket (`~/.freshell/managed-runtime/control/supervisor.sock`) is LIVE — connect-probe it as in the strict sequence, because the supervisor unit @@ -867,7 +871,7 @@ PY done if [ -n "${D:-}" ]; then case "$D" in - /tmp/tmp.*) ;; + "${TMPDIR:-/tmp}"/tmp.*) ;; # mktemp honors $TMPDIR when set *) echo "REFUSING: D=$D is not a mktemp records directory"; exit 1 ;; esac if [ "${RECOVERY_MINTED:-0}" != "1" ]; then @@ -879,7 +883,10 @@ PY # its self-consistent SOUL_ID/TID) with another run's WS would # otherwise stop the old run's soul and delete the new workspace: if [ -n "${WS:-}" ] && [ -f "$D/run.env" ]; then - RECORDED_WS=$(sed -n 's/^WS=//p' "$D/run.env" | tail -1) + # Source the file to UNESCAPE the %q-encoded values — a raw + # text read would compare escaped bytes against the real path + # and false-refuse any path with spaces: + RECORDED_WS=$(bash -c '. "$1"; printf "%s" "${WS:-}"' _ "$D/run.env") if [ -n "$RECORDED_WS" ] && [ "$RECORDED_WS" != "$WS" ]; then echo "REFUSING: D=$D records workspace $RECORDED_WS, but WS=$WS — mixed runs; resolve manually" exit 1 From 7864992661dce7831b118e9d5b41fbd77377b9c7 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 15:50:34 -0700 Subject: [PATCH 46/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2044?= =?UTF-8?q?=20=E2=80=94=20deployment=20checkout=20established=20before=20t?= =?UTF-8?q?he=20build=20(units=20run=20their=20own=20paths,=20not=20the=20?= =?UTF-8?q?cwd);=20settings=20restore=20is=20conditional=20on=20the=20prob?= =?UTF-8?q?e=20path=20still=20being=20current;=20BWS=20%-quoted=20in=20run?= =?UTF-8?q?.env?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 26 ++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 95aad1242..4b38d15d6 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -88,8 +88,17 @@ client, one opt-in feature flag, and two systemd user units. A server-only rebuild or restart is NOT enough to change soul behavior — and a plain rebuild silently loses it. -**Build surface.** Build all three binaries from the repo root when any of -them changes: +**Establish the deployment checkout FIRST.** The units run binaries from +THEIR configured paths — not from whatever directory you happen to be +in. Read them (`systemctl --user show freshell-rust.service -p +WorkingDirectory --value` and the supervisor unit's `ExecStart` path, +typically `%h/code/freshell`), and run the build from THAT checkout. +Artifacts built anywhere else (a dev worktree, a scratch clone) are +never started by the units — a mistake you would otherwise discover only +after restarting production. + +**Build surface.** Build all three binaries from the deployment checkout +when any of them changes: ```bash cargo build --release -p freshell-server -p freshell-supervisor \ @@ -684,7 +693,7 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T esac BWS=$(mktemp -d "$PROBE_ROOT/bws.XXXXXX") # unique per run chmod 0770 "$BWS" # same in-soul-shell access requirement as $WS - echo "BWS=$BWS" >> "$D/run.env" + printf 'BWS=%q\n' "$BWS" >> "$D/run.env" # The directory picker PERSISTS the chosen path as OpenCode's default # cwd (the confirm handler patches settings) — record the current # default so step 8 can restore it: @@ -1017,6 +1026,16 @@ print("" if cur is None else cur)') # re-reading it — curl succeeds as a shell command even on a 4xx/5xx: if [ -n "${BWS:-}" ] && [ -f "$D/old-opencode-cwd.txt" ]; then OLD=$(cat "$D/old-opencode-cwd.txt") + # Restore ONLY if the setting still points at THIS probe's path — + # a newer user choice made after our read must not be overwritten: + CURNOW=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ + | python3 -c 'import json,sys +d=json.load(sys.stdin); d=d.get("data",d) +cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") +print("" if cur is None else cur)') + if [ "$CURNOW" != "$BWS" ]; then + echo "OpenCode default no longer points at the probe path (now '$CURNOW') — leaving the newer value alone" + else BODY=$(python3 -c 'import json,sys old=sys.argv[1] or None print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":old}}}}))' "$OLD") @@ -1032,6 +1051,7 @@ cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") cur="" if cur is None else cur sys.exit(0 if cur==sys.argv[1] else 1)' "$OLD" \ || { echo "settings restore did not stick (expected OpenCode cwd '$OLD') — fix Settings → Coding CLI → OpenCode manually BEFORE deleting the probe dirs"; exit 1; } + fi elif [ -n "${BWS:-}" ]; then # RECOVERY MODE: the original $D (with the saved pre-check default) # was lost with the shell, so the original value is unrecoverable. From 78c680761036b520e1dcfc89ac22152d0a987708 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 16:00:56 -0700 Subject: [PATCH 47/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2045?= =?UTF-8?q?=20=E2=80=94=20concurrency=20parse=20handles=20quoted=20values?= =?UTF-8?q?=20and=20leading=20zeroes=20(base-10=20arithmetic=20after=20dig?= =?UTF-8?q?it=20validation);=20picker-save=20wait=20branches=20instead=20o?= =?UTF-8?q?f=20wedging=20cleanup=20(newer=20user=20choices=20are=20preserv?= =?UTF-8?q?ed,=20not=20blocking)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 21 +++++++++++++------ 1 file changed, 15 insertions(+), 6 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 4b38d15d6..f6bf77335 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -253,7 +253,15 @@ the procedure mid-way and leave the service down. Two sequences: V=$(sed -n 's/^FRESHELL_RUNTIME_STARTUP_CONCURRENCY=//p' "$EF" 2>/dev/null | tail -1) if [ -n "$V" ]; then C="$V"; fi done + # Systemd EnvironmentFile values may be QUOTED ("1"), and bash + # arithmetic treats leading zeroes as OCTAL (08 is an error) — strip + # quotes, validate digits, then force base 10: + case "$C" in + \"*\") C=${C#\"}; C=${C%\"} ;; + \'*\') C=${C#\'}; C=${C%\'} ;; + esac case "$C" in ''|*[!0-9]*) C=4;; esac # absent/garbage -> supervisor default + C=$((10#$C)) # defeat octal interpretation [ "$C" -ge 1 ] 2>/dev/null || C=1 # supervisor clamps to 1..16 — mirror it [ "$C" -le 16 ] 2>/dev/null || C=16 # Count UNIQUE soul IDs: the inventory stores per-incarnation rows, so @@ -1012,12 +1020,13 @@ print("" if cur is None else cur)') if [ "$CUR" = "$BWS" ]; then break; fi sleep 2 done - if [ "$CUR" != "$BWS" ]; then - # The save state is unknown — a still-in-flight picker save could - # re-point the default AFTER our restore, into a deleted dir. Do - # not proceed; wait for the save to settle and re-run cleanup: - echo "OpenCode default never settled on the probe path — the picker's async save may still be in flight; wait, then re-run this step" - exit 1 + # Branch on what the default settled to — cleanup is NEVER wedged: + # $BWS -> the picker's save landed; the restore below puts + # back the recorded default. + # $OLD -> already the desired value; nothing to do. + # anything else -> a NEWER user choice won the race; leave it. + if [ "$CUR" != "$BWS" ] && [ -f "$D/old-opencode-cwd.txt" ] && [ "$CUR" != "$(cat "$D/old-opencode-cwd.txt")" ]; then + echo "OpenCode default was changed to a newer value ('$CUR') after the probe — leaving it untouched" fi fi # Then the restore itself — JSON-safe (a path containing From 7433b777e6d6c522d1067855894fc3a3a8dc4682 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 16:10:11 -0700 Subject: [PATCH 48/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2046?= =?UTF-8?q?=20=E2=80=94=20verified=20HTTP+shape=20reads=20at=20every=20set?= =?UTF-8?q?tings=20site;=20late=20picker=20saves=20swept=20after=20workspa?= =?UTF-8?q?ce=20deletion;=20header=20scrub=20moved=20after=20the=20sweep?= =?UTF-8?q?=20(it=20needs=20the=20token);=20residual=20no-CAS=20window=20s?= =?UTF-8?q?tated=20honestly?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 72 +++++++++++++++---- 1 file changed, 59 insertions(+), 13 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index f6bf77335..777fd1a62 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -1012,11 +1012,18 @@ print("stopped browser soul:", o["outcome"])' # about-to-be-deleted workspace: if [ -n "${BWS:-}" ]; then for i in $(seq 1 15); do - CUR=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ - | python3 -c 'import json,sys -d=json.load(sys.stdin); d=d.get("data",d) -cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") -print("" if cur is None else cur)') + SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ + -o "$D/settings.json" -w '%{http_code}' -H @"$D/hdr" \ + http://127.0.0.1:3001/api/settings) + if ! grep -q '^2' <<<"$SETTINGS_CODE"; then sleep 2; continue; fi + CUR=$(python3 -c 'import json,sys +d=json.load(open(sys.argv[1])) +assert isinstance(d, dict), "malformed settings response" +d=d.get("data",d) +cc=d.get("codingCli") +assert isinstance(cc, dict), "malformed settings response: no codingCli" +cur=(cc.get("providers") or {}).get("opencode",{}).get("cwd") +print("" if cur is None else cur)' "$D/settings.json" 2>/dev/null) || { sleep 2; continue; } if [ "$CUR" = "$BWS" ]; then break; fi sleep 2 done @@ -1037,11 +1044,18 @@ print("" if cur is None else cur)') OLD=$(cat "$D/old-opencode-cwd.txt") # Restore ONLY if the setting still points at THIS probe's path — # a newer user choice made after our read must not be overwritten: - CURNOW=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ - | python3 -c 'import json,sys -d=json.load(sys.stdin); d=d.get("data",d) -cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") -print("" if cur is None else cur)') + SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ + -o "$D/settings.json" -w '%{http_code}' -H @"$D/hdr" \ + http://127.0.0.1:3001/api/settings) + grep -q '^2' <<<"$SETTINGS_CODE" || { echo "settings read failed (HTTP $SETTINGS_CODE)"; exit 1; } + CURNOW=$(python3 -c 'import json,sys +d=json.load(open(sys.argv[1])) +assert isinstance(d, dict), "malformed settings response" +d=d.get("data",d) +cc=d.get("codingCli") +assert isinstance(cc, dict), "malformed settings response: no codingCli" +cur=(cc.get("providers") or {}).get("opencode",{}).get("cwd") +print("" if cur is None else cur)' "$D/settings.json") if [ "$CURNOW" != "$BWS" ]; then echo "OpenCode default no longer points at the probe path (now '$CURNOW') — leaving the newer value alone" else @@ -1081,13 +1095,37 @@ print("" if cur is None else cur)') echo "NOTE: the pre-check OpenCode default was lost with the lost shell — set your preferred default in Settings → Coding CLI → OpenCode" fi fi - rm -f "${RUN:-/nonexistent}/hdr" "$D/hdr" # scrub NOW; the trap only - # fires at shell EXIT, which is too late for $RUN/hdr in a still- - # living interactive shell # Delete the probe workspaces, and the ROOT only when it holds # nothing besides this run's known workspaces — a root with other # entries (another workspace, hidden data) is NOT ours to delete: rm -rf "$D" "$WS" "${BWS:-}" + # POST-DELETION SWEEP: the client's settings save is unawaited, so a + # late landing can set the default to the now-DELETED $BWS after all + # the pre-restore checks passed. Give it a moment, re-read (verified), + # and clear the default if it landed on the probe path: + if [ -n "${BWS:-}" ]; then + sleep 3 + SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ + -o "/tmp/freshell-settings-sweep.json" -w '%{http_code}' -H @"$RUN/hdr" \ + http://127.0.0.1:3001/api/settings 2>/dev/null) || SETTINGS_CODE=000 + if grep -q '^2' <<<"$SETTINGS_CODE" 2>/dev/null; then + SWEEP_CWD=$(python3 -c 'import json,sys +d=json.load(open(sys.argv[1])); d=d.get("data",d) +cur=((d.get("codingCli") or {}).get("providers",{}).get("opencode",{}) or {}).get("cwd") +print("" if cur is None else cur)' /tmp/freshell-settings-sweep.json 2>/dev/null) || SWEEP_CWD="" + if [ "$SWEEP_CWD" = "$BWS" ]; then + BODY=$(python3 -c 'import json;print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":None}}}}))') + curl -s --connect-timeout 5 --max-time 30 -X PATCH -H @"$RUN/hdr" \ + -H "Content-Type: application/json" -d "$BODY" \ + http://127.0.0.1:3001/api/settings > /dev/null + echo "NOTE: a late picker save re-pointed OpenCode at the deleted probe dir — cleared; set your default in Settings" + fi + rm -f /tmp/freshell-settings-sweep.json + fi + fi + rm -f "${RUN:-/nonexistent}/hdr" "$D/hdr" 2>/dev/null # scrub last — + # the sweep above needed $RUN/hdr; the trap alone only fires at shell + # EXIT, which is too late in a still-living interactive shell if [ -n "${PROBE_ROOT:-}" ]; then UNEXPECTED="" shopt -s nullglob @@ -1112,6 +1150,14 @@ print("" if cur is None else cur)') documented recheck-and-retry; delete `$D`, `$WS`, and `$BWS` manually once BOTH probe souls are resolved. + HONEST LIMIT on the settings restore: the settings API has no + compare-and-swap, so a concurrent writer can still race the + read-then-PATCH sequence; the procedure narrows the window (re-read + immediately before patching, verified reads, and the post-deletion + sweep above) but cannot eliminate a truly concurrent write. The sweep + catches saves that land during cleanup; a save landing after the + sweep's read is out of the runbook's hands — Settings shows it. + Do NOT `docker stop` a probe container directly — the supervisor treats an externally stopped container as an unexpected exit and runs its automatic recovery path, which may recreate the soul or mark a From 94c1d04e8667fcfd7c063374faa477f4a3ff8743 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 16:15:49 -0700 Subject: [PATCH 49/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2047?= =?UTF-8?q?=20=E2=80=94=20the=20two=20remaining=20settings=20reads=20are?= =?UTF-8?q?=20verified=20(step-7=20capture,=20recovery);=20sweep=20uses=20?= =?UTF-8?q?a=20fresh=20private=20dir=20(settings=20responses=20carry=20sec?= =?UTF-8?q?rets),=20re-mints=20its=20token=20after=20a=20lost=20shell,=20a?= =?UTF-8?q?nd=20checks=20the=20final=20PATCH?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 95 +++++++++++++------ 1 file changed, 67 insertions(+), 28 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 777fd1a62..01dce4bec 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -704,13 +704,21 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T printf 'BWS=%q\n' "$BWS" >> "$D/run.env" # The directory picker PERSISTS the chosen path as OpenCode's default # cwd (the confirm handler patches settings) — record the current - # default so step 8 can restore it: - curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ - | python3 -c 'import json,sys -d=json.load(sys.stdin) + # default (VERIFIED read: an error response must not masquerade as + # an empty saved default): + SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ + -o "$D/settings.json" -w '%{http_code}' -H @"$D/hdr" \ + http://127.0.0.1:3001/api/settings) + grep -q '^2' <<<"$SETTINGS_CODE" || { echo "settings read failed (HTTP $SETTINGS_CODE)"; exit 1; } + python3 -c 'import json,sys +d=json.load(open(sys.argv[1])) +assert isinstance(d, dict), "malformed settings response" d=d.get("data",d) -print((d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") or "")' \ - > "$D/old-opencode-cwd.txt" +cc=d.get("codingCli") +assert isinstance(cc, dict), "malformed settings response: no codingCli" +cur=(cc.get("providers") or {}).get("opencode",{}).get("cwd") +print("" if cur is None else cur)' "$D/settings.json" > "$D/old-opencode-cwd.txt" + rm -f "$D/settings.json" # the response can carry secrets — do not keep it # Hard-refresh the web client, then in the UI: open the pane picker # and choose OPENCODE specifically — the only managed-enabled # coding-CLI provider in the current release (Claude, Codex, and @@ -1080,12 +1088,21 @@ sys.exit(0 if cur==sys.argv[1] else 1)' "$OLD" \ # was lost with the shell, so the original value is unrecoverable. # If the current default still points at $BWS, clearing it is the # only safe option — otherwise deleting $BWS would leave OpenCode - # defaulting into a deleted directory: - CUR=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/settings \ - | python3 -c 'import json,sys -d=json.load(sys.stdin); d=d.get("data",d) -cur=(d.get("codingCli") or {}).get("providers",{}).get("opencode",{}).get("cwd") -print("" if cur is None else cur)') + # defaulting into a deleted directory (VERIFIED read: an error + # response must not masquerade as a changed default): + SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ + -o "$D/settings.json" -w '%{http_code}' -H @"$D/hdr" \ + http://127.0.0.1:3001/api/settings) + grep -q '^2' <<<"$SETTINGS_CODE" || { echo "settings read failed (HTTP $SETTINGS_CODE)"; exit 1; } + CUR=$(python3 -c 'import json,sys +d=json.load(open(sys.argv[1])) +assert isinstance(d, dict), "malformed settings response" +d=d.get("data",d) +cc=d.get("codingCli") +assert isinstance(cc, dict), "malformed settings response: no codingCli" +cur=(cc.get("providers") or {}).get("opencode",{}).get("cwd") +print("" if cur is None else cur)' "$D/settings.json") + rm -f "$D/settings.json" # the response can carry secrets if [ "$CUR" = "$BWS" ]; then BODY=$(python3 -c 'import json;print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":None}}}}))') CODE=$(curl -s --connect-timeout 5 --max-time 30 -o "$D/settings-clear.out" -w '%{http_code}' -X PATCH \ @@ -1101,27 +1118,49 @@ print("" if cur is None else cur)') rm -rf "$D" "$WS" "${BWS:-}" # POST-DELETION SWEEP: the client's settings save is unawaited, so a # late landing can set the default to the now-DELETED $BWS after all - # the pre-restore checks passed. Give it a moment, re-read (verified), - # and clear the default if it landed on the probe path: + # the pre-restore checks passed. The sweep reads into a FRESH PRIVATE + # dir (the response can carry secrets; a fixed /tmp path would be + # shared between concurrent runs and left behind on failure) and + # needs a live token header — after a lost shell, mint both: if [ -n "${BWS:-}" ]; then sleep 3 - SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ - -o "/tmp/freshell-settings-sweep.json" -w '%{http_code}' -H @"$RUN/hdr" \ - http://127.0.0.1:3001/api/settings 2>/dev/null) || SETTINGS_CODE=000 - if grep -q '^2' <<<"$SETTINGS_CODE" 2>/dev/null; then - SWEEP_CWD=$(python3 -c 'import json,sys + SWEEP_DIR=$(mktemp -d) + umask 077 + if [ ! -f "${RUN:-/nonexistent}/hdr" ]; then + SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) + TOK="" + if [ "${SPID:-0}" -gt 0 ] 2>/dev/null; then + TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p' || true) + fi + if [ -n "$TOK" ]; then + printf 'x-auth-token: %s\n' "$TOK" > "$SWEEP_DIR/hdr" + fi + fi + HDR="${RUN:-$SWEEP_DIR}/hdr" + if [ -f "$HDR" ]; then + SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ + -o "$SWEEP_DIR/settings.json" -w '%{http_code}' -H @"$HDR" \ + http://127.0.0.1:3001/api/settings) || SETTINGS_CODE=000 + if grep -q '^2' <<<"$SETTINGS_CODE"; then + SWEEP_CWD=$(python3 -c 'import json,sys d=json.load(open(sys.argv[1])); d=d.get("data",d) -cur=((d.get("codingCli") or {}).get("providers",{}).get("opencode",{}) or {}).get("cwd") -print("" if cur is None else cur)' /tmp/freshell-settings-sweep.json 2>/dev/null) || SWEEP_CWD="" - if [ "$SWEEP_CWD" = "$BWS" ]; then - BODY=$(python3 -c 'import json;print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":None}}}}))') - curl -s --connect-timeout 5 --max-time 30 -X PATCH -H @"$RUN/hdr" \ - -H "Content-Type: application/json" -d "$BODY" \ - http://127.0.0.1:3001/api/settings > /dev/null - echo "NOTE: a late picker save re-pointed OpenCode at the deleted probe dir — cleared; set your default in Settings" +cc=d.get("codingCli") +assert isinstance(cc, dict), "malformed settings response" +cur=((cc.get("providers") or {}).get("opencode",{}) or {}).get("cwd") +print("" if cur is None else cur)' "$SWEEP_DIR/settings.json") || SWEEP_CWD="" + if [ "$SWEEP_CWD" = "$BWS" ]; then + BODY=$(python3 -c 'import json;print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":None}}}}))') + PATCH_CODE=$(curl -s --connect-timeout 5 --max-time 30 \ + -o "$SWEEP_DIR/patch.out" -w '%{http_code}' -X PATCH -H @"$HDR" \ + -H "Content-Type: application/json" -d "$BODY" \ + http://127.0.0.1:3001/api/settings) + grep -q '^2' <<<"$PATCH_CODE" \ + || { echo "late-save clear FAILED (HTTP $PATCH_CODE) — clear Settings → Coding CLI → OpenCode manually"; rm -rf "$SWEEP_DIR"; exit 1; } + echo "NOTE: a late picker save re-pointed OpenCode at the deleted probe dir — cleared; set your default in Settings" + fi fi - rm -f /tmp/freshell-settings-sweep.json fi + rm -rf "$SWEEP_DIR" fi rm -f "${RUN:-/nonexistent}/hdr" "$D/hdr" 2>/dev/null # scrub last — # the sweep above needed $RUN/hdr; the trap alone only fires at shell From ffffd69a64dcaa70c97795d07620f1c38e2133bd Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 16:20:36 -0700 Subject: [PATCH 50/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2048?= =?UTF-8?q?=20=E2=80=94=20sweep=20re-mints=20the=20token=20from=20both=20s?= =?UTF-8?q?ources=20(environ=20then=20.env)=20and=20fails=20closed=20inste?= =?UTF-8?q?ad=20of=20skipping;=20malformed=20sweep=20responses=20abort=20w?= =?UTF-8?q?ith=20manual-resolution=20instructions;=20guard=20pins=20worksp?= =?UTF-8?q?ace=20depth=20(case=20*=20matches=20slashes,=20nested=20paths?= =?UTF-8?q?=20refused)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 72 +++++++++++++------ 1 file changed, 52 insertions(+), 20 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 01dce4bec..a09e7ea29 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -876,6 +876,13 @@ PY "$HOME"/freshell-probe.*) ;; *) echo "REFUSING: $V=$VAL is not a freshell-probe workspace — resolve manually"; exit 1 ;; esac + # A case-pattern `*` matches `/` too — pin the depth: the parent's + # own parent must be EXACTLY the home, so a path nested deeper + # under a probe-named directory cannot pass: + if [ "$(dirname "$PARENT")" != "$HOME" ]; then + echo "REFUSING: $V=$VAL is nested below $PARENT — workspaces sit directly under the probe root" + exit 1 + fi # A NAME pattern is not ownership: a real workspace could sit under # a freshell-probe-named directory. Probe workspaces are created # EMPTY and stay empty (in-soul provider state lives in the soul @@ -1126,39 +1133,64 @@ print("" if cur is None else cur)' "$D/settings.json") sleep 3 SWEEP_DIR=$(mktemp -d) umask 077 - if [ ! -f "${RUN:-/nonexistent}/hdr" ]; then + HDR="${RUN:-/nonexistent}/hdr" + if [ ! -f "$HDR" ]; then + # Re-mint the token with the SAME two-source extraction the rest + # of the procedure uses — a server that loaded its token from + # .env never exposes it in /proc: SPID=$(systemctl --user show freshell-rust.service -p MainPID --value) TOK="" if [ "${SPID:-0}" -gt 0 ] 2>/dev/null; then TOK=$(tr '\0' '\n' < "/proc/$SPID/environ" | sed -n 's/^AUTH_TOKEN=//p' || true) fi + if [ -z "$TOK" ]; then + SWEEP_CHECKOUT=$(systemctl --user show freshell-rust.service -p WorkingDirectory --value) + TOK=$(python3 - "$SWEEP_CHECKOUT/.env" <<'PY' +import os,re,sys +path=sys.argv[1] +if not os.path.exists(path): sys.exit(0) +last=None +for line in open(path): + m=re.match(r'^\s*(?:export\s+)?AUTH_TOKEN\s*=\s*(.*)$', line.rstrip("\n")) + if m: last=m.group(1) +if last is None: sys.exit(0) +if last[:1] in ('"',"'"): + end=last.find(last[0],1) + print(last[1:end] if end>0 else last[1:]) +else: + print(re.split(r'\s#', last.strip(), 1)[0].strip()) +PY +) + fi if [ -n "$TOK" ]; then printf 'x-auth-token: %s\n' "$TOK" > "$SWEEP_DIR/hdr" + HDR="$SWEEP_DIR/hdr" fi fi - HDR="${RUN:-$SWEEP_DIR}/hdr" - if [ -f "$HDR" ]; then - SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ - -o "$SWEEP_DIR/settings.json" -w '%{http_code}' -H @"$HDR" \ - http://127.0.0.1:3001/api/settings) || SETTINGS_CODE=000 - if grep -q '^2' <<<"$SETTINGS_CODE"; then - SWEEP_CWD=$(python3 -c 'import json,sys + # The sweep is the LAST verification gate — it fails closed: + [ -f "$HDR" ] \ + || { echo "sweep cannot run (no token) — check Settings → Coding CLI → OpenCode manually; the default may point at the deleted probe dir"; rm -rf "$SWEEP_DIR"; exit 1; } + SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ + -o "$SWEEP_DIR/settings.json" -w '%{http_code}' -H @"$HDR" \ + http://127.0.0.1:3001/api/settings) || SETTINGS_CODE=000 + grep -q '^2' <<<"$SETTINGS_CODE" \ + || { echo "sweep settings read FAILED (HTTP $SETTINGS_CODE) — check Settings → Coding CLI → OpenCode manually"; rm -rf "$SWEEP_DIR"; exit 1; } + SWEEP_CWD=$(python3 -c 'import json,sys d=json.load(open(sys.argv[1])); d=d.get("data",d) cc=d.get("codingCli") assert isinstance(cc, dict), "malformed settings response" cur=((cc.get("providers") or {}).get("opencode",{}) or {}).get("cwd") -print("" if cur is None else cur)' "$SWEEP_DIR/settings.json") || SWEEP_CWD="" - if [ "$SWEEP_CWD" = "$BWS" ]; then - BODY=$(python3 -c 'import json;print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":None}}}}))') - PATCH_CODE=$(curl -s --connect-timeout 5 --max-time 30 \ - -o "$SWEEP_DIR/patch.out" -w '%{http_code}' -X PATCH -H @"$HDR" \ - -H "Content-Type: application/json" -d "$BODY" \ - http://127.0.0.1:3001/api/settings) - grep -q '^2' <<<"$PATCH_CODE" \ - || { echo "late-save clear FAILED (HTTP $PATCH_CODE) — clear Settings → Coding CLI → OpenCode manually"; rm -rf "$SWEEP_DIR"; exit 1; } - echo "NOTE: a late picker save re-pointed OpenCode at the deleted probe dir — cleared; set your default in Settings" - fi - fi +print("" if cur is None else cur)' "$SWEEP_DIR/settings.json") \ + || { echo "sweep got a malformed settings response — check Settings → Coding CLI → OpenCode manually"; rm -rf "$SWEEP_DIR"; exit 1; } + if [ "$SWEEP_CWD" = "$BWS" ]; then + BODY=$(python3 -c 'import json;print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":None}}}}))') + PATCH_CODE=$(curl -s --connect-timeout 5 --max-time 30 \ + -o "$SWEEP_DIR/patch.out" -w '%{http_code}' -X PATCH -H @"$HDR" \ + -H "Content-Type: application/json" -d "$BODY" \ + http://127.0.0.1:3001/api/settings) + grep -q '^2' <<<"$PATCH_CODE" \ + || { echo "late-save clear FAILED (HTTP $PATCH_CODE) — clear Settings → Coding CLI → OpenCode manually"; rm -rf "$SWEEP_DIR"; exit 1; } + echo "NOTE: a late picker save re-pointed OpenCode at the deleted probe dir — cleared; set your default in Settings" fi rm -rf "$SWEEP_DIR" fi From f0fe6ba32ea23d4a1861a63f303a07b478e2bc4b Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 16:36:55 -0700 Subject: [PATCH 51/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2049?= =?UTF-8?q?=20=E2=80=94=20docker=20socket=20and=20registry=20root=20derive?= =?UTF-8?q?d=20from=20the=20installed=20supervisor=20unit=20(specifiers=20?= =?UTF-8?q?expanded,=20site-specific=20units=20honored);=20allowed-roots?= =?UTF-8?q?=20fallback=20for=20the=20browser=20probe=20when=20allowedFileP?= =?UTF-8?q?aths=20excludes=20home;=20guard=20ownership=20keyed=20to=20the?= =?UTF-8?q?=20RECORDED=20run=20root=20(records=20path)=20with=20the=20home?= =?UTF-8?q?=20pattern=20only=20for=20recovery;=20four=20branch=20fixtures?= =?UTF-8?q?=20pass?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 97 +++++++++++++++---- 1 file changed, 80 insertions(+), 17 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index a09e7ea29..d946a830a 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -117,17 +117,28 @@ post-deploy soul sweep: ```bash set -euo pipefail RUN=$(mktemp -d) # private per-run dir for the survival snapshots -# Pin the Docker CLI to the SAME daemon socket the supervisor uses (its -# unit passes --docker-socket %t/docker.sock, the user runtime dir). A -# machine with a second Docker daemon would otherwise capture against -# the wrong one and the survival check would pass vacuously. DOCKER_CONTEXT -# must go too — it takes precedence over DOCKER_HOST: +# Pin the Docker CLI to the SAME daemon socket the supervisor's unit +# actually names (its --docker-socket argument, with systemd specifiers +# %t = the user runtime dir and %h = the home expanded) — a site-specific +# unit may point elsewhere, and a machine with a second Docker daemon +# would otherwise capture against the wrong one, passing the survival +# check vacuously. DOCKER_CONTEXT must go too — it precedes DOCKER_HOST: unset DOCKER_CONTEXT || true -export DOCKER_HOST="unix://${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/docker.sock" +UNIT_ARGS=$(systemctl --user cat freshell-supervisor.service | tr ' ' '\n') +DSOCK=$(sed -n 's/^--docker-socket=\?//p' <<<"$UNIT_ARGS" | head -1) +DSOCK=${DSOCK//%t/${XDG_RUNTIME_DIR:-/run/user/$(id -u)}} +DSOCK=${DSOCK//%h/$HOME} +DSOCK=${DSOCK:-${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/docker.sock} +export DOCKER_HOST="unix://$DSOCK" +# Same for the registry root the soul inventory reads (used below): +REGROOT=$(sed -n 's/^--registry-root=\?//p' <<<"$UNIT_ARGS" | head -1) +REGROOT=${REGROOT//%t/${XDG_RUNTIME_DIR:-/run/user/$(id -u)}} +REGROOT=${REGROOT//%h/$HOME} +REGROOT=${REGROOT:-$HOME/.freshell/managed-runtime/registry} # Scope the capture to THIS installation: the daemon may host containers # from other Freshell installations with the same name shape, and the # installation label is the only discriminator: -INSTALL_ID=$(sqlite3 "$HOME/.freshell/managed-runtime/registry/runtime.sqlite3" \ +INSTALL_ID=$(sqlite3 "$REGROOT/runtime.sqlite3" \ "select installation_id from installation") test -n "$INSTALL_ID" || { echo "cannot read installation id from the registry"; exit 1; } echo "$INSTALL_ID" > "$RUN/installation-id" @@ -187,7 +198,7 @@ else # REGISTRY is readable directly, and it is the authority on desired # state. Read the baseline from sqlite so a desired-running soul with # no running container is not invisible to every later check: - sqlite3 "$HOME/.freshell/managed-runtime/registry/runtime.sqlite3" \ + sqlite3 "$REGROOT/runtime.sqlite3" \ "select soul_id from souls where desired_state='running'" \ > "$RUN/desired-running.txt" 2>/dev/null \ || { echo "cannot read the registry sqlite — locate desired-running souls manually"; exit 1; } @@ -727,7 +738,31 @@ print("" if cur is None else cur)' "$D/settings.json" > "$D/old-opencode-cwd.txt # correctly deployed client). A plain shell pane will NOT do either: # shell panes skip the directory step and always take the default # workspace. OpenCode's directory step has a free-text PATH INPUT: - # type the absolute $BWS path there and confirm. Then WAIT (bounded): + # type the absolute $BWS path there and confirm. If the picker + # REJECTS the path (the directory is validated against the + # `allowedFilePaths` setting — a deployment whose allowed roots + # exclude $HOME will 403 the probe root), mint the probe root under + # an allowed root instead and record it so the cleanup guard + # validates against the RECORDED root: + ALLOWED=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" \ + http://127.0.0.1:3001/api/settings \ + | python3 -c 'import json,sys +d=json.load(sys.stdin); d=d.get("data",d) +for r in (d.get("allowedFilePaths") or []): + print(r)') + if ! printf '%s\n' "$ALLOWED" | grep -q "^$HOME"; then + FIRST_ALLOWED=$(head -1 <<<"$ALLOWED") + if [ -n "$FIRST_ALLOWED" ]; then + PROBE_ROOT=$(mktemp -d "$FIRST_ALLOWED/freshell-probe.XXXXXX") + chmod 0770 "$PROBE_ROOT" + printf 'PROBE_ROOT=%q\n' "$PROBE_ROOT" >> "$D/run.env" + rm -rf "$BWS" + BWS=$(mktemp -d "$PROBE_ROOT/bws.XXXXXX") + chmod 0770 "$BWS" + printf 'BWS=%q\n' "$BWS" >> "$D/run.env" + fi + fi + # Then WAIT (bounded): # the confirmation dispatches a UI update, terminal.create goes out # from a React effect, and the managed launch is asynchronous — so # the soul may not exist the instant you confirm. Poll up to ~120s: @@ -862,24 +897,52 @@ PY GUARD_ROOT="" if [ -n "${PROBE_ROOT:-}" ]; then PROBE_ROOT=$(realpath -e "$PROBE_ROOT") || { echo "PROBE_ROOT: path does not exist"; exit 1; } - case "$PROBE_ROOT" in - "$HOME"/freshell-probe.*) GUARD_ROOT="$PROBE_ROOT" ;; - *) echo "REFUSING: PROBE_ROOT=$PROBE_ROOT is not a freshell-probe run root"; exit 1 ;; - esac + # Ownership when records exist: the root must be the one THIS run + # recorded in run.env (which may live under an allowed root other + # than $HOME); without records (recovery), fall back to the + # $HOME/freshell-probe.* pattern: + if [ -f "${D:-/nonexistent}/run.env" ]; then + RECORDED_ROOT=$(bash -c '. "$1"; printf "%s" "${PROBE_ROOT:-}"' _ "${D:-}/run.env") + if [ -n "$RECORDED_ROOT" ] && [ "$RECORDED_ROOT" != "$PROBE_ROOT" ]; then + echo "REFUSING: PROBE_ROOT=$PROBE_ROOT is not the root this run recorded ($RECORDED_ROOT)" + exit 1 + fi + case "$(basename "$PROBE_ROOT")" in + freshell-probe.*) GUARD_ROOT="$PROBE_ROOT" ;; + *) echo "REFUSING: PROBE_ROOT=$PROBE_ROOT is not a freshell-probe run root"; exit 1 ;; + esac + else + case "$PROBE_ROOT" in + "$HOME"/freshell-probe.*) GUARD_ROOT="$PROBE_ROOT" ;; + *) echo "REFUSING: PROBE_ROOT=$PROBE_ROOT is not a freshell-probe run root"; exit 1 ;; + esac + fi fi for V in WS BWS; do eval "VAL=\${$V:-}" [ -n "$VAL" ] || continue VAL=$(realpath -e "$VAL") || { echo "$V: path does not exist"; exit 1; } PARENT=$(dirname "$VAL") + if [ -n "$GUARD_ROOT" ]; then + # Records exist: the workspace must sit directly under the root + # this run recorded: + if [ "$PARENT" != "$GUARD_ROOT" ]; then + echo "REFUSING: $V=$VAL is not under this run's probe root $GUARD_ROOT — resolve manually" + exit 1 + fi + else case "$PARENT" in "$HOME"/freshell-probe.*) ;; *) echo "REFUSING: $V=$VAL is not a freshell-probe workspace — resolve manually"; exit 1 ;; esac - # A case-pattern `*` matches `/` too — pin the depth: the parent's - # own parent must be EXACTLY the home, so a path nested deeper - # under a probe-named directory cannot pass: - if [ "$(dirname "$PARENT")" != "$HOME" ]; then + fi + # A case-pattern `*` matches `/` too — pin the depth IN THE + # PATTERN BRANCH ONLY (records-path workspaces already proved + # their parent is the recorded root, which may sit under an + # allowed root away from $HOME): the parent's own parent must be + # EXACTLY the home, so a path nested deeper under a probe-named + # directory cannot pass: + if [ -z "$GUARD_ROOT" ] && [ "$(dirname "$PARENT")" != "$HOME" ]; then echo "REFUSING: $V=$VAL is nested below $PARENT — workspaces sit directly under the probe root" exit 1 fi From a4ae60ede31593a12479d7db0fdc731c67189540 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 16:45:18 -0700 Subject: [PATCH 52/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2050?= =?UTF-8?q?=20=E2=80=94=20flag=20values=20extracted=20for=20both=20--flag?= =?UTF-8?q?=20value=20and=20--flag=3Dvalue=20forms;=20browser=20fallback?= =?UTF-8?q?=20uses=20a=20separate=20recorded=20BROWSER=5FROOT=20(WS=20stay?= =?UTF-8?q?s=20under=20PROBE=5FROOT,=20no=20shared-root=20rule);=20recover?= =?UTF-8?q?y=20anchors=20by=20root=20basename=20(allowed-roots=20deploymen?= =?UTF-8?q?ts=20clean=20up=20too);=20control=20socket=20derived=20from=20t?= =?UTF-8?q?he=20installed=20unit;=20guard=20branches=20fixture-verified?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 85 ++++++++++++------- 1 file changed, 54 insertions(+), 31 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index d946a830a..f4d726fda 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -124,14 +124,17 @@ RUN=$(mktemp -d) # private per-run dir for the survival snapshots # would otherwise capture against the wrong one, passing the survival # check vacuously. DOCKER_CONTEXT must go too — it precedes DOCKER_HOST: unset DOCKER_CONTEXT || true -UNIT_ARGS=$(systemctl --user cat freshell-supervisor.service | tr ' ' '\n') -DSOCK=$(sed -n 's/^--docker-socket=\?//p' <<<"$UNIT_ARGS" | head -1) +# Extract flag values with a lookbehind that accepts BOTH the +# `--flag value` and `--flag=value` forms (splitting on spaces would +# separate the flag from its value and yield empty matches): +UNIT_CAT=$(systemctl --user cat freshell-supervisor.service) +DSOCK=$(grep -oP '(?<=--docker-socket[= ])\S+' <<<"$UNIT_CAT" | head -1) DSOCK=${DSOCK//%t/${XDG_RUNTIME_DIR:-/run/user/$(id -u)}} DSOCK=${DSOCK//%h/$HOME} DSOCK=${DSOCK:-${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/docker.sock} export DOCKER_HOST="unix://$DSOCK" # Same for the registry root the soul inventory reads (used below): -REGROOT=$(sed -n 's/^--registry-root=\?//p' <<<"$UNIT_ARGS" | head -1) +REGROOT=$(grep -oP '(?<=--registry-root[= ])\S+' <<<"$UNIT_CAT" | head -1) REGROOT=${REGROOT//%t/${XDG_RUNTIME_DIR:-/run/user/$(id -u)}} REGROOT=${REGROOT//%h/$HOME} REGROOT=${REGROOT:-$HOME/.freshell/managed-runtime/registry} @@ -238,7 +241,13 @@ the procedure mid-way and leave the service down. Two sequences: ```bash systemctl --user start freshell-supervisor.service - SOCK="$HOME/.freshell/managed-runtime/control/supervisor.sock" + # The control socket path comes from the INSTALLED unit (a + # site-specific --control-socket must not defeat this wait): + SOCK=$(grep -oP '(?<=--control-socket[= ])\S+' \ + <(systemctl --user cat freshell-supervisor.service) | head -1) + SOCK=${SOCK//%t/${XDG_RUNTIME_DIR:-/run/user/$(id -u)}} + SOCK=${SOCK//%h/$HOME} + SOCK=${SOCK:-$HOME/.freshell/managed-runtime/control/supervisor.sock} # Two failure detectors: supervisor-unit LIVENESS (fast — a dead unit # ends the wait immediately) and an inventory-sized absolute CAP # (stall — a live-but-wedged process must not hang the redeploy @@ -753,11 +762,13 @@ for r in (d.get("allowedFilePaths") or []): if ! printf '%s\n' "$ALLOWED" | grep -q "^$HOME"; then FIRST_ALLOWED=$(head -1 <<<"$ALLOWED") if [ -n "$FIRST_ALLOWED" ]; then - PROBE_ROOT=$(mktemp -d "$FIRST_ALLOWED/freshell-probe.XXXXXX") - chmod 0770 "$PROBE_ROOT" - printf 'PROBE_ROOT=%q\n' "$PROBE_ROOT" >> "$D/run.env" + # Keep PROBE_ROOT (and the REST probe's WS) where it is — mint a + # SEPARATE browser root under the allowed root and record it: + BROWSER_ROOT=$(mktemp -d "$FIRST_ALLOWED/freshell-probe.XXXXXX") + chmod 0770 "$BROWSER_ROOT" + printf 'BROWSER_ROOT=%q\n' "$BROWSER_ROOT" >> "$D/run.env" rm -rf "$BWS" - BWS=$(mktemp -d "$PROBE_ROOT/bws.XXXXXX") + BWS=$(mktemp -d "$BROWSER_ROOT/bws.XXXXXX") chmod 0770 "$BWS" printf 'BWS=%q\n' "$BWS" >> "$D/run.env" fi @@ -849,8 +860,8 @@ print("UI pane is managed:", rows[-1]["soulId"])' "$BWS" eval "VAL=\${$V:-}" [ -n "$VAL" ] || continue VAL=$(realpath -e "$VAL") || { echo "$V: path does not exist"; exit 1; } - case "$VAL" in - "$HOME"/freshell-probe.*) printf -v "$V" '%s' "$VAL" ;; + case "$(basename "$(dirname "$VAL")")" in + freshell-probe.*) printf -v "$V" '%s' "$VAL" ;; *) echo "REFUSING: $V=$VAL is not under a freshell-probe run root — resolve manually"; exit 1 ;; esac done @@ -923,29 +934,33 @@ PY [ -n "$VAL" ] || continue VAL=$(realpath -e "$VAL") || { echo "$V: path does not exist"; exit 1; } PARENT=$(dirname "$VAL") - if [ -n "$GUARD_ROOT" ]; then - # Records exist: the workspace must sit directly under the root - # this run recorded: - if [ "$PARENT" != "$GUARD_ROOT" ]; then - echo "REFUSING: $V=$VAL is not under this run's probe root $GUARD_ROOT — resolve manually" + if [ -n "${GUARD_ROOT:-}" ]; then + # Records exist: WS must sit under the recorded PROBE_ROOT and + # BWS under the recorded BROWSER_ROOT (when the allowed-roots + # fallback minted one) — per-variable roots, never a shared one: + if [ -f "${D:-/nonexistent}/run.env" ]; then + case "$V" in + WS) WANT_ROOT=$(bash -c '. "$1"; printf "%s" "${PROBE_ROOT:-}"' _ "${D:-}/run.env") ;; + BWS) WANT_ROOT=$(bash -c '. "$1"; printf "%s" "${BROWSER_ROOT:-${PROBE_ROOT:-}}"' _ "${D:-}/run.env") ;; + esac + fi + WANT_ROOT="${WANT_ROOT:-$GUARD_ROOT}" + if [ "$PARENT" != "$WANT_ROOT" ]; then + echo "REFUSING: $V=$VAL is not under this run's recorded root for it ($WANT_ROOT) — resolve manually" exit 1 fi else - case "$PARENT" in - "$HOME"/freshell-probe.*) ;; - *) echo "REFUSING: $V=$VAL is not a freshell-probe workspace — resolve manually"; exit 1 ;; + # Recovery (no records): the workspace's parent must BE a + # freshell-probe root by BASENAME — anchored anywhere (an + # allowed-roots deployment puts the browser probe outside $HOME). + # The emptiness check and the interactive confirm below still + # apply; a real directory named freshell-probe.* is the residual + # risk the confirm step exists to catch: + case "$(basename "$PARENT")" in + freshell-probe.*) ;; + *) echo "REFUSING: $V=$VAL is not under a freshell-probe run root — resolve manually"; exit 1 ;; esac fi - # A case-pattern `*` matches `/` too — pin the depth IN THE - # PATTERN BRANCH ONLY (records-path workspaces already proved - # their parent is the recorded root, which may sit under an - # allowed root away from $HOME): the parent's own parent must be - # EXACTLY the home, so a path nested deeper under a probe-named - # directory cannot pass: - if [ -z "$GUARD_ROOT" ] && [ "$(dirname "$PARENT")" != "$HOME" ]; then - echo "REFUSING: $V=$VAL is nested below $PARENT — workspaces sit directly under the probe root" - exit 1 - fi # A NAME pattern is not ownership: a real workspace could sit under # a freshell-probe-named directory. Probe workspaces are created # EMPTY and stay empty (in-soul provider state lives in the soul @@ -956,11 +971,13 @@ PY echo "REFUSING: $V=$VAL is not empty — probe workspaces are empty; resolve manually" exit 1 fi + # The records path above already tied each workspace to its OWN + # recorded root (PROBE_ROOT for WS, BROWSER_ROOT for BWS) — no + # cross-variable sharing rule applies; the recovery path anchors + # by basename above. Only remember the first parent for any later + # recovery-mode checks: if [ -z "$GUARD_ROOT" ]; then GUARD_ROOT="$PARENT" - elif [ "$PARENT" != "$GUARD_ROOT" ]; then - echo "REFUSING: $V=$VAL belongs to a different probe root than the other paths — resolve manually" - exit 1 fi printf -v "$V" '%s' "$VAL" done @@ -1276,6 +1293,12 @@ print("" if cur is None else cur)' "$SWEEP_DIR/settings.json") \ echo "probe root kept — it contains unexpected entries:$UNEXPECTED" fi fi + # The allowed-roots fallback's browser root (when this run minted + # one and it is empty after the workspace deletion): + if [ -n "${BROWSER_ROOT:-}" ] && [ -d "$BROWSER_ROOT" ] \ + && [ -z "$(find "$BROWSER_ROOT" -mindepth 1 -print -quit 2>/dev/null)" ]; then + rm -rf "$BROWSER_ROOT" + fi ``` If the block aborts before `verified_empty`, the token header is From 66e25f5f8216fcd8ec5f186532ba14fa18cd2bf1 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 16:54:35 -0700 Subject: [PATCH 53/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2051?= =?UTF-8?q?=20=E2=80=94=20allowed-path=20check=20tests=20whether=20the=20p?= =?UTF-8?q?robe=20path=20itself=20is=20permitted=20(prefix=20match=20again?= =?UTF-8?q?st=20each=20entry,=20not=20a=20^HOME=20grep);=20flag=20extracti?= =?UTF-8?q?on=20takes=20the=20LAST=20match=20(drop-ins=20that=20reset=20Ex?= =?UTF-8?q?ecStart=20come=20after=20the=20base=20unit=20in=20systemctl=20c?= =?UTF-8?q?at)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 20 +++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index f4d726fda..28c590736 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -128,13 +128,13 @@ unset DOCKER_CONTEXT || true # `--flag value` and `--flag=value` forms (splitting on spaces would # separate the flag from its value and yield empty matches): UNIT_CAT=$(systemctl --user cat freshell-supervisor.service) -DSOCK=$(grep -oP '(?<=--docker-socket[= ])\S+' <<<"$UNIT_CAT" | head -1) +DSOCK=$(grep -oP '(?<=--docker-socket[= ])\S+' <<<"$UNIT_CAT" | tail -1) # tail: a drop-in that resets ExecStart comes LAST in cat output DSOCK=${DSOCK//%t/${XDG_RUNTIME_DIR:-/run/user/$(id -u)}} DSOCK=${DSOCK//%h/$HOME} DSOCK=${DSOCK:-${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/docker.sock} export DOCKER_HOST="unix://$DSOCK" # Same for the registry root the soul inventory reads (used below): -REGROOT=$(grep -oP '(?<=--registry-root[= ])\S+' <<<"$UNIT_CAT" | head -1) +REGROOT=$(grep -oP '(?<=--registry-root[= ])\S+' <<<"$UNIT_CAT" | tail -1) # tail: drop-in resets win REGROOT=${REGROOT//%t/${XDG_RUNTIME_DIR:-/run/user/$(id -u)}} REGROOT=${REGROOT//%h/$HOME} REGROOT=${REGROOT:-$HOME/.freshell/managed-runtime/registry} @@ -244,7 +244,7 @@ the procedure mid-way and leave the service down. Two sequences: # The control socket path comes from the INSTALLED unit (a # site-specific --control-socket must not defeat this wait): SOCK=$(grep -oP '(?<=--control-socket[= ])\S+' \ - <(systemctl --user cat freshell-supervisor.service) | head -1) + <(systemctl --user cat freshell-supervisor.service) | tail -1) # tail: a drop-in that resets ExecStart comes LAST SOCK=${SOCK//%t/${XDG_RUNTIME_DIR:-/run/user/$(id -u)}} SOCK=${SOCK//%h/$HOME} SOCK=${SOCK:-$HOME/.freshell/managed-runtime/control/supervisor.sock} @@ -759,7 +759,19 @@ print("" if cur is None else cur)' "$D/settings.json" > "$D/old-opencode-cwd.txt d=json.load(sys.stdin); d=d.get("data",d) for r in (d.get("allowedFilePaths") or []): print(r)') - if ! printf '%s\n' "$ALLOWED" | grep -q "^$HOME"; then + # Is the probe path itself PERMITTED? An allowed entry A permits the + # probe path when the path starts with A — e.g. allowed=["/home/dan/work"] + # does NOT permit $HOME/freshell-probe.* even though the entry sits + # under $HOME; allowed=["/home"] DOES. A grep for "^$HOME" answers + # the wrong question for both cases: + PROBE_PATH_OK=0 + while read -r A; do + [ -n "$A" ] || continue + case "$HOME/freshell-probe.probe" in + "$A"*) PROBE_PATH_OK=1; break ;; + esac + done <<<"$ALLOWED" + if [ "$PROBE_PATH_OK" != 1 ]; then FIRST_ALLOWED=$(head -1 <<<"$ALLOWED") if [ -n "$FIRST_ALLOWED" ]; then # Keep PROBE_ROOT (and the REST probe's WS) where it is — mint a From 5bb84ef65c02b1eaa793b30cec0b9183bde370c9 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:04:23 -0700 Subject: [PATCH 54/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2052?= =?UTF-8?q?=20=E2=80=94=20allowed-path=20check=20mirrors=20the=20server=20?= =?UTF-8?q?(/=20boundary=20or=20exact=20match,=20~=20normalized);=20allowe?= =?UTF-8?q?d-list=20read=20is=20HTTP-=20and=20shape-verified;=20step-7=20c?= =?UTF-8?q?hmod=20pinned=20to=20depth+basename=20(case=20*=20matches=20sla?= =?UTF-8?q?shes)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 39 +++++++++++++------ 1 file changed, 28 insertions(+), 11 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 28c590736..11012f03d 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -715,10 +715,16 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T # pattern check: PROBE_ROOT=$(realpath -e "$PROBE_ROOT") \ || { echo "PROBE_ROOT: path does not exist"; exit 1; } - case "$PROBE_ROOT" in - "$HOME"/freshell-probe.*) chmod 0770 "$PROBE_ROOT" ;; - *) echo "REFUSING: PROBE_ROOT=$PROBE_ROOT is not a freshell-probe run root"; exit 1 ;; - esac + # Pin depth and basename — a case-pattern `*` matches `/` too, so a + # bare "$HOME"/freshell-probe.* would chmod an inherited nested + # directory like $HOME/freshell-probe.old/project: + if [ "$(dirname "$PROBE_ROOT")" = "$HOME" ] \ + && case "$(basename "$PROBE_ROOT")" in freshell-probe.*) true ;; *) false ;; esac; then + chmod 0770 "$PROBE_ROOT" + else + echo "REFUSING: PROBE_ROOT=$PROBE_ROOT is not a freshell-probe run root directly under home" + exit 1 + fi BWS=$(mktemp -d "$PROBE_ROOT/bws.XXXXXX") # unique per run chmod 0770 "$BWS" # same in-soul-shell access requirement as $WS printf 'BWS=%q\n' "$BWS" >> "$D/run.env" @@ -753,12 +759,19 @@ print("" if cur is None else cur)' "$D/settings.json" > "$D/old-opencode-cwd.txt # exclude $HOME will 403 the probe root), mint the probe root under # an allowed root instead and record it so the cleanup guard # validates against the RECORDED root: - ALLOWED=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" \ - http://127.0.0.1:3001/api/settings \ - | python3 -c 'import json,sys -d=json.load(sys.stdin); d=d.get("data",d) -for r in (d.get("allowedFilePaths") or []): - print(r)') + SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ + -o "$D/settings.json" -w '%{http_code}' -H @"$D/hdr" \ + http://127.0.0.1:3001/api/settings) + grep -q '^2' <<<"$SETTINGS_CODE" || { echo "settings read failed (HTTP $SETTINGS_CODE)"; exit 1; } + ALLOWED=$(python3 -c 'import json,sys +d=json.load(open(sys.argv[1])) +assert isinstance(d, dict), "malformed settings response" +d=d.get("data",d) +afp=d.get("allowedFilePaths") +assert afp is None or isinstance(afp, list), "malformed allowedFilePaths" +for r in (afp or []): + print(r)' "$D/settings.json") || { echo "malformed settings response"; exit 1; } + rm -f "$D/settings.json" # the response can carry secrets # Is the probe path itself PERMITTED? An allowed entry A permits the # probe path when the path starts with A — e.g. allowed=["/home/dan/work"] # does NOT permit $HOME/freshell-probe.* even though the entry sits @@ -767,8 +780,12 @@ for r in (d.get("allowedFilePaths") or []): PROBE_PATH_OK=0 while read -r A; do [ -n "$A" ] || continue + A=${A/#\~/$HOME} # the server normalizes ~ roots — mirror it + # The server's is_path_allowed requires the probe path to BE the + # root or sit under it at a / boundary — a bare string prefix + # would wrongly accept /home/dan/freshell for /home/dan/freshell-probe.x: case "$HOME/freshell-probe.probe" in - "$A"*) PROBE_PATH_OK=1; break ;; + "$A"|"$A"/*) PROBE_PATH_OK=1; break ;; esac done <<<"$ALLOWED" if [ "$PROBE_PATH_OK" != 1 ]; then From 545431a50f3951738d46f9b13ea3b3eaedaee792 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:13:16 -0700 Subject: [PATCH 55/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2053?= =?UTF-8?q?=20=E2=80=94=20allowed=20check=20tests=20this=20run's=20actual?= =?UTF-8?q?=20BWS;=20~=20expanded=20in=20the=20allowed=20root=20before=20q?= =?UTF-8?q?uoted=20mktemp;=20sweep=20dir=20scrubbed=20by=20the=20EXIT=20tr?= =?UTF-8?q?ap=20on=20every=20exit=20path?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/development/self-hosted-launch-runbook.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 11012f03d..e90357464 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -781,15 +781,17 @@ for r in (afp or []): while read -r A; do [ -n "$A" ] || continue A=${A/#\~/$HOME} # the server normalizes ~ roots — mirror it - # The server's is_path_allowed requires the probe path to BE the - # root or sit under it at a / boundary — a bare string prefix - # would wrongly accept /home/dan/freshell for /home/dan/freshell-probe.x: - case "$HOME/freshell-probe.probe" in + # Test THIS run's actual $BWS, not a stand-in path: the server's + # is_path_allowed requires the path to BE the root or sit under it + # at a / boundary — a bare string prefix would wrongly accept + # /home/dan/freshell for /home/dan/freshell-probe.x: + case "$BWS" in "$A"|"$A"/*) PROBE_PATH_OK=1; break ;; esac done <<<"$ALLOWED" if [ "$PROBE_PATH_OK" != 1 ]; then FIRST_ALLOWED=$(head -1 <<<"$ALLOWED") + FIRST_ALLOWED=${FIRST_ALLOWED/#\~/$HOME} # expand ~ — quoted mktemp will not if [ -n "$FIRST_ALLOWED" ]; then # Keep PROBE_ROOT (and the REST probe's WS) where it is — mint a # SEPARATE browser root under the allowed root and record it: @@ -926,7 +928,7 @@ PY test -n "$TOK" || { echo "AUTH_TOKEN not found — extract it manually"; exit 1; } umask 077; printf 'x-auth-token: %s\n' "$TOK" > "$D/hdr" fi - trap 'rm -f "${RUN:-/nonexistent}/hdr" "${D:-/nonexistent}/hdr"' EXIT # BOTH headers, every exit path, set -u-safe + trap 'rm -f "${RUN:-/nonexistent}/hdr" "${D:-/nonexistent}/hdr"; [ -n "${SWEEP_DIR:-}" ] && rm -rf "${SWEEP_DIR:-/nonexistent}"' EXIT # both headers AND any sweep dir minted below (set -u-safe) # DESTRUCTIVE-PATH GUARD (both the normal and recovery flows pass # through here). Canonicalize every probe path and tie them together: # PROBE_ROOT must BE a freshell-probe run root; WS and BWS must be From 0bbfc23ff74224e1621961f019f876af611004c5 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:22:12 -0700 Subject: [PATCH 56/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2054?= =?UTF-8?q?=20=E2=80=94=20non-Linux=20allowed=20roots=20skipped=20(server?= =?UTF-8?q?=20maps=20WSL=20Windows=20roots=20itself);=20post-deletion=20sw?= =?UTF-8?q?eep=20is=20a=20bounded=20poll;=20run.env=20records=20PROBE=5FRO?= =?UTF-8?q?OT=20and=20BWS=20is=20cross-checked=20against=20it;=20settings?= =?UTF-8?q?=20responses=20deleted=20immediately=20and=20the=20retained-rec?= =?UTF-8?q?ords=20prose=20no=20longer=20claims=20they=20are=20non-secret?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 77 +++++++++++++------ 1 file changed, 53 insertions(+), 24 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index e90357464..2eec46cd3 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -627,7 +627,7 @@ PY # ${VAR:-} defaults keep the trap functional under set -u even in a # lost-shell recovery where RUN is unset): trap 'rm -f "${RUN:-/nonexistent}/hdr" "${D:-/nonexistent}/hdr"' EXIT - printf 'D=%q\nWS=%q\nCHECKOUT=%q\n' "$D" "$WS" "$CHECKOUT" > "$D/run.env" + printf 'D=%q\nWS=%q\nCHECKOUT=%q\nPROBE_ROOT=%q\n' "$D" "$WS" "$CHECKOUT" "$PROBE_ROOT" > "$D/run.env" # create the probe pane (example); record terminalId from the response: curl -s --connect-timeout 5 --max-time 30 -X POST -H @"$D/hdr" -H "Content-Type: application/json" \ -d '{"mode":"shell","cwd":"'"$WS"'","name":"home-mount probe"}' \ @@ -790,8 +790,16 @@ for r in (afp or []): esac done <<<"$ALLOWED" if [ "$PROBE_PATH_OK" != 1 ]; then - FIRST_ALLOWED=$(head -1 <<<"$ALLOWED") - FIRST_ALLOWED=${FIRST_ALLOWED/#\~/$HOME} # expand ~ — quoted mktemp will not + # Pick the first USABLE Linux-path root: the server maps + # Windows-style roots (WSL deployments, e.g. C:\Users\...) to + # Linux paths itself — a literal such entry would break mktemp: + FIRST_ALLOWED="" + while read -r A; do + A=${A/#\~/$HOME} + case "$A" in + /*) FIRST_ALLOWED="$A"; break ;; + esac + done <<<"$ALLOWED" if [ -n "$FIRST_ALLOWED" ]; then # Keep PROBE_ROOT (and the REST probe's WS) where it is — mint a # SEPARATE browser root under the allowed root and record it: @@ -1025,14 +1033,25 @@ PY # TIE the records to THIS run's workspace: mixing one run's D (and # its self-consistent SOUL_ID/TID) with another run's WS would # otherwise stop the old run's soul and delete the new workspace: - if [ -n "${WS:-}" ] && [ -f "$D/run.env" ]; then - # Source the file to UNESCAPE the %q-encoded values — a raw - # text read would compare escaped bytes against the real path - # and false-refuse any path with spaces: - RECORDED_WS=$(bash -c '. "$1"; printf "%s" "${WS:-}"' _ "$D/run.env") - if [ -n "$RECORDED_WS" ] && [ "$RECORDED_WS" != "$WS" ]; then - echo "REFUSING: D=$D records workspace $RECORDED_WS, but WS=$WS — mixed runs; resolve manually" - exit 1 + # Source the file to UNESCAPE the %q-encoded values — a raw + # text read would compare escaped bytes against the real path + # and false-refuse any path with spaces. BOTH recorded paths are + # cross-checked, so a mistaken BWS pointing at another empty + # workspace is refused: + if [ -f "$D/run.env" ]; then + if [ -n "${WS:-}" ]; then + RECORDED_WS=$(bash -c '. "$1"; printf "%s" "${WS:-}"' _ "$D/run.env") + if [ -n "$RECORDED_WS" ] && [ "$RECORDED_WS" != "$WS" ]; then + echo "REFUSING: D=$D records workspace $RECORDED_WS, but WS=$WS — mixed runs; resolve manually" + exit 1 + fi + fi + if [ -n "${BWS:-}" ]; then + RECORDED_BWS=$(bash -c '. "$1"; printf "%s" "${BWS:-}"' _ "$D/run.env") + if [ -n "$RECORDED_BWS" ] && [ "$RECORDED_BWS" != "$BWS" ]; then + echo "REFUSING: D=$D records browser workspace $RECORDED_BWS, but BWS=$BWS — mixed runs; resolve manually" + exit 1 + fi fi fi fi @@ -1149,7 +1168,8 @@ d=d.get("data",d) cc=d.get("codingCli") assert isinstance(cc, dict), "malformed settings response: no codingCli" cur=(cc.get("providers") or {}).get("opencode",{}).get("cwd") -print("" if cur is None else cur)' "$D/settings.json" 2>/dev/null) || { sleep 2; continue; } +print("" if cur is None else cur)' "$D/settings.json" 2>/dev/null) || { rm -f "$D/settings.json"; sleep 2; continue; } + rm -f "$D/settings.json" # may carry secrets if [ "$CUR" = "$BWS" ]; then break; fi sleep 2 done @@ -1182,6 +1202,7 @@ cc=d.get("codingCli") assert isinstance(cc, dict), "malformed settings response: no codingCli" cur=(cc.get("providers") or {}).get("opencode",{}).get("cwd") print("" if cur is None else cur)' "$D/settings.json") + rm -f "$D/settings.json" # may carry secrets — remove immediately if [ "$CURNOW" != "$BWS" ]; then echo "OpenCode default no longer points at the probe path (now '$CURNOW') — leaving the newer value alone" else @@ -1241,6 +1262,8 @@ print("" if cur is None else cur)' "$D/settings.json") # shared between concurrent runs and left behind on failure) and # needs a live token header — after a lost shell, mint both: if [ -n "${BWS:-}" ]; then + # Bounded post-deletion POLL rather than a single read — the + # client's save queue can deliver late: sleep 3 SWEEP_DIR=$(mktemp -d) umask 077 @@ -1281,19 +1304,23 @@ PY # The sweep is the LAST verification gate — it fails closed: [ -f "$HDR" ] \ || { echo "sweep cannot run (no token) — check Settings → Coding CLI → OpenCode manually; the default may point at the deleted probe dir"; rm -rf "$SWEEP_DIR"; exit 1; } - SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ - -o "$SWEEP_DIR/settings.json" -w '%{http_code}' -H @"$HDR" \ - http://127.0.0.1:3001/api/settings) || SETTINGS_CODE=000 - grep -q '^2' <<<"$SETTINGS_CODE" \ - || { echo "sweep settings read FAILED (HTTP $SETTINGS_CODE) — check Settings → Coding CLI → OpenCode manually"; rm -rf "$SWEEP_DIR"; exit 1; } - SWEEP_CWD=$(python3 -c 'import json,sys + SWEEP_HITS=0 + for SWEEP_I in $(seq 1 6); do + SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ + -o "$SWEEP_DIR/settings.json" -w '%{http_code}' -H @"$HDR" \ + http://127.0.0.1:3001/api/settings) || SETTINGS_CODE=000 + if ! grep -q '^2' <<<"$SETTINGS_CODE"; then sleep 3; continue; fi + SWEEP_CWD=$(python3 -c 'import json,sys d=json.load(open(sys.argv[1])); d=d.get("data",d) cc=d.get("codingCli") assert isinstance(cc, dict), "malformed settings response" cur=((cc.get("providers") or {}).get("opencode",{}) or {}).get("cwd") -print("" if cur is None else cur)' "$SWEEP_DIR/settings.json") \ - || { echo "sweep got a malformed settings response — check Settings → Coding CLI → OpenCode manually"; rm -rf "$SWEEP_DIR"; exit 1; } - if [ "$SWEEP_CWD" = "$BWS" ]; then +print("" if cur is None else cur)' "$SWEEP_DIR/settings.json") || { sleep 3; continue; } + if [ "$SWEEP_CWD" = "$BWS" ]; then SWEEP_HITS=1; break; fi + sleep 3 + done + rm -f "$SWEEP_DIR/settings.json" + if [ "$SWEEP_HITS" = 1 ]; then BODY=$(python3 -c 'import json;print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":None}}}}))') PATCH_CODE=$(curl -s --connect-timeout 5 --max-time 30 \ -o "$SWEEP_DIR/patch.out" -w '%{http_code}' -X PATCH -H @"$HDR" \ @@ -1333,9 +1360,11 @@ print("" if cur is None else cur)' "$SWEEP_DIR/settings.json") \ ``` If the block aborts before `verified_empty`, the token header is - already gone (the `trap` removed it) while the non-secret probe - records — the soul ID and the tab response — remain in `$D` for the - documented recheck-and-retry; delete `$D`, `$WS`, and `$BWS` + already gone (the `trap` removed it) while the probe records — the + soul ID and the tab response — remain in `$D` for the documented + recheck-and-retry. Settings responses are deleted immediately after + each parse (they can carry API keys); anything that still remains is + inside a 0700 mktemp directory. Delete `$D`, `$WS`, and `$BWS` manually once BOTH probe souls are resolved. HONEST LIMIT on the settings restore: the settings API has no From 5e41f0469c6fdc143d1b0d451e2eb745b80024a0 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:31:10 -0700 Subject: [PATCH 57/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2055?= =?UTF-8?q?=20=E2=80=94=20sweep=20fails=20closed=20when=20every=20read=20f?= =?UTF-8?q?ails;=20recovery=20validates=20each=20workspace=20independently?= =?UTF-8?q?=20(no=20cross-root=20sharing);=20Windows-only=20allowed=20list?= =?UTF-8?q?s=20prompt=20for=20the=20Linux=20path=20of=20an=20allowed=20roo?= =?UTF-8?q?t;=20recorded=20roots=20canonicalized=20(symlinked=20roots=20co?= =?UTF-8?q?mpare=20correctly);=20secret-hygiene=20prose=20accurate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 41 +++++++++++++------ 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 2eec46cd3..6a69edd47 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -800,6 +800,18 @@ for r in (afp or []): /*) FIRST_ALLOWED="$A"; break ;; esac done <<<"$ALLOWED" + if [ -z "$FIRST_ALLOWED" ]; then + # Windows-only allowed lists (WSL): the server maps such roots + # itself and this script cannot replicate the mapping — ask the + # operator for the Linux form of one allowed root (visible in + # the picker) and mint under that: + read -r -p "Allowed roots are Windows-style; type the LINUX path of one allowed root for the browser probe: " FIRST_ALLOWED + FIRST_ALLOWED=${FIRST_ALLOWED/#\~/$HOME} + case "$FIRST_ALLOWED" in + /*) ;; + *) echo "not a Linux absolute path — aborting"; exit 1 ;; + esac + fi if [ -n "$FIRST_ALLOWED" ]; then # Keep PROBE_ROOT (and the REST probe's WS) where it is — mint a # SEPARATE browser root under the allowed root and record it: @@ -953,6 +965,7 @@ PY # $HOME/freshell-probe.* pattern: if [ -f "${D:-/nonexistent}/run.env" ]; then RECORDED_ROOT=$(bash -c '. "$1"; printf "%s" "${PROBE_ROOT:-}"' _ "${D:-}/run.env") + RECORDED_ROOT=${RECORDED_ROOT:+$(realpath -m "$RECORDED_ROOT")} if [ -n "$RECORDED_ROOT" ] && [ "$RECORDED_ROOT" != "$PROBE_ROOT" ]; then echo "REFUSING: PROBE_ROOT=$PROBE_ROOT is not the root this run recorded ($RECORDED_ROOT)" exit 1 @@ -984,6 +997,9 @@ PY esac fi WANT_ROOT="${WANT_ROOT:-$GUARD_ROOT}" + # Canonicalize the recorded root — a symlinked allowed root + # records uncanonicalized while $VAL above went through realpath: + [ -n "$WANT_ROOT" ] && WANT_ROOT=$(realpath -m "$WANT_ROOT") if [ "$PARENT" != "$WANT_ROOT" ]; then echo "REFUSING: $V=$VAL is not under this run's recorded root for it ($WANT_ROOT) — resolve manually" exit 1 @@ -1010,14 +1026,10 @@ PY echo "REFUSING: $V=$VAL is not empty — probe workspaces are empty; resolve manually" exit 1 fi - # The records path above already tied each workspace to its OWN - # recorded root (PROBE_ROOT for WS, BROWSER_ROOT for BWS) — no - # cross-variable sharing rule applies; the recovery path anchors - # by basename above. Only remember the first parent for any later - # recovery-mode checks: - if [ -z "$GUARD_ROOT" ]; then - GUARD_ROOT="$PARENT" - fi + # Each workspace is validated independently: the records path ties + # it to its OWN recorded root (PROBE_ROOT for WS, BROWSER_ROOT for + # BWS); recovery anchors by basename. No cross-variable sharing — + # the two roots are legitimately different: printf -v "$V" '%s' "$VAL" done if [ -n "${D:-}" ]; then @@ -1304,7 +1316,7 @@ PY # The sweep is the LAST verification gate — it fails closed: [ -f "$HDR" ] \ || { echo "sweep cannot run (no token) — check Settings → Coding CLI → OpenCode manually; the default may point at the deleted probe dir"; rm -rf "$SWEEP_DIR"; exit 1; } - SWEEP_HITS=0 + SWEEP_HITS=0; SWEEP_READS_OK=0 for SWEEP_I in $(seq 1 6); do SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ -o "$SWEEP_DIR/settings.json" -w '%{http_code}' -H @"$HDR" \ @@ -1316,10 +1328,14 @@ cc=d.get("codingCli") assert isinstance(cc, dict), "malformed settings response" cur=((cc.get("providers") or {}).get("opencode",{}) or {}).get("cwd") print("" if cur is None else cur)' "$SWEEP_DIR/settings.json") || { sleep 3; continue; } + SWEEP_READS_OK=$((SWEEP_READS_OK + 1)) if [ "$SWEEP_CWD" = "$BWS" ]; then SWEEP_HITS=1; break; fi sleep 3 done rm -f "$SWEEP_DIR/settings.json" + # Fail closed: a sweep where EVERY read failed verified nothing: + [ "$SWEEP_READS_OK" -ge 1 ] \ + || { echo "sweep could not read settings at all — check Settings → Coding CLI → OpenCode manually; the default may point at the deleted probe dir"; rm -rf "$SWEEP_DIR"; exit 1; } if [ "$SWEEP_HITS" = 1 ]; then BODY=$(python3 -c 'import json;print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":None}}}}))') PATCH_CODE=$(curl -s --connect-timeout 5 --max-time 30 \ @@ -1362,9 +1378,10 @@ print("" if cur is None else cur)' "$SWEEP_DIR/settings.json") || { sleep 3; con If the block aborts before `verified_empty`, the token header is already gone (the `trap` removed it) while the probe records — the soul ID and the tab response — remain in `$D` for the documented - recheck-and-retry. Settings responses are deleted immediately after - each parse (they can carry API keys); anything that still remains is - inside a 0700 mktemp directory. Delete `$D`, `$WS`, and `$BWS` + recheck-and-retry. Settings responses are deleted right after + each single-read parse; the sweep keeps its response only inside its + private loop directory and removes it with the directory (responses + can carry API keys). Delete `$D`, `$WS`, and `$BWS` manually once BOTH probe souls are resolved. HONEST LIMIT on the settings restore: the settings API has no From 1a43c95cb6716c86fea4f4b3b433962cb1bb49b1 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:40:35 -0700 Subject: [PATCH 58/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2056?= =?UTF-8?q?=20=E2=80=94=20empty=20allowedFilePaths=20is=20permit-all;=20qu?= =?UTF-8?q?ick-sequence=20baseline=20re-captured=20after=20the=20build=20(?= =?UTF-8?q?souls=20created=20during=20the=20build=20are=20baselined);=20st?= =?UTF-8?q?ale-client=20step-7=20failure=20gets=20an=20explicit=20cleanup?= =?UTF-8?q?=20path=20(BROWSER=5FCHECK=5FFAILED=3D1);=20recovery=20confirms?= =?UTF-8?q?=20the=20browser=20soul=20separately?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 44 +++++++++++++++---- 1 file changed, 35 insertions(+), 9 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 6a69edd47..42074930b 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -105,8 +105,11 @@ cargo build --release -p freshell-server -p freshell-supervisor \ -p freshell-session-host --features freshell-server/managed-runtime-v1 ``` -**Preliminary: capture the baseline.** Before restarting anything, -capture two snapshots into a private per-run directory (fixed `/tmp` +**Preliminary: capture the baseline.** Capture two snapshots into a +private per-run directory (in the QUICK sequence, re-run this block +AFTER the build and immediately before the restart, so souls created +during the build are baselined; in the STRICT sequence, run it before +the stop) (fixed `/tmp` paths would let a second run overwrite the first run's baseline before its comparison): the RUNNING containers (their IDs and exact start times) and the DESIRED-RUNNING soul IDs — the supervisor reconciles @@ -222,9 +225,13 @@ process tree (for example an SSH session to the host) — a host-side Freshell terminal pane is itself killed by the restart, which would abort the procedure mid-way and leave the service down. Two sequences: -- Quick (compatible changes): build, then restart immediately, accepting - the short skew window. -- Strict (changes that must never skew): FIRST `systemctl --user stop +- Quick (compatible changes): build, then RE-CAPTURE the baseline block + immediately before the restart (souls created during the build are + otherwise unbaselined and their loss goes undetected), then restart. + The short skew window is accepted. +- Strict (changes that must never skew): FIRST re-run the baseline + capture block above (nothing new can be created after the stop, so + a pre-stop baseline is complete), then `systemctl --user stop freshell-supervisor.service` (the server stops with it via the `Requires=` coupling), THEN run the build command from the Build surface above — never before the stop, or the running supervisor @@ -777,6 +784,11 @@ for r in (afp or []): # does NOT permit $HOME/freshell-probe.* even though the entry sits # under $HOME; allowed=["/home"] DOES. A grep for "^$HOME" answers # the wrong question for both cases: + # An empty/unset allowedFilePaths means the server permits ALL + # paths — treat that as allowed, not denied: + if [ -z "$(printf '%s' "$ALLOWED" | tr -d '[:space:]')" ]; then + PROBE_PATH_OK=1 + else PROBE_PATH_OK=0 while read -r A; do [ -n "$A" ] || continue @@ -789,6 +801,7 @@ for r in (afp or []): "$A"|"$A"/*) PROBE_PATH_OK=1; break ;; esac done <<<"$ALLOWED" + fi if [ "$PROBE_PATH_OK" != 1 ]; then # Pick the first USABLE Linux-path root: the server maps # Windows-style roots (WSL deployments, e.g. C:\Users\...) to @@ -1135,12 +1148,25 @@ if rows: else: print("")' "$BWS") if [ -z "$BS" ]; then - # Step 7 VERIFIED a running soul in this workspace; a row that - # vanished since is an anomaly, not success — do not delete the - # workspace while the soul's intent is unresolved: - { echo "browser-check soul disappeared from the registry before cleanup — investigate before deleting $BWS"; exit 1; } + # Two cases: step 7 FAILED with a diagnosed stale client (no + # managed soul was ever created — export BROWSER_CHECK_FAILED=1 + # to clean up), or a previously verified soul vanished (an + # anomaly — refuse): + if [ "${BROWSER_CHECK_FAILED:-0}" = "1" ]; then + echo "no managed browser soul (stale-client step 7) — restoring settings and removing the workspace" + else + { echo "browser-check soul disappeared from the registry before cleanup — investigate before deleting $BWS (or export BROWSER_CHECK_FAILED=1 if step 7 failed with a stale client)"; exit 1; } + fi fi read -r BS_ID BS_REV BS_STATE BS_CID <<< "$BS" + # In recovery (no records tie BWS to this run), confirm the + # browser soul separately — mixing two runs' empty probe dirs + # must not stop another run's soul unconfirmed: + if [ "${RECOVERY_MINTED:-0}" = "1" ]; then + echo "Recovery matched browser soul: $BS_ID state=$BS_STATE workspace=$BWS" + read -r -p "Stop and delete this BROWSER probe soul too? Type 'yes' to confirm: " BCONFIRM + [ "$BCONFIRM" = "yes" ] || { echo "browser probe left running — resolve $BWS manually"; exit 1; } + fi # An EXITED incarnation can still be desired-running (eligible for # recovery) — only a committed stop intent makes it safe to delete # the workspace, so require desiredState=="stopped" too: From fcad476f9316c217fa30363e0ff3463a41c2b369 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:50:02 -0700 Subject: [PATCH 59/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2057?= =?UTF-8?q?=20=E2=80=94=20stale-client=20cleanup=20skips=20the=20stop=20ma?= =?UTF-8?q?chinery=20cleanly=20(no=20empty-ID=20requests,=20confirm=20supp?= =?UTF-8?q?ressed);=20baseline=20re-capture=20reuses=20RUN=20in=20the=20sa?= =?UTF-8?q?me=20shell=20(an=20orphaned=20run=20dir=20would=20leak=20its=20?= =?UTF-8?q?token=20header)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/development/self-hosted-launch-runbook.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 42074930b..5c5fc2295 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -119,7 +119,11 @@ post-deploy soul sweep: ```bash set -euo pipefail -RUN=$(mktemp -d) # private per-run dir for the survival snapshots +# REUSE $RUN when re-running this block in the same shell (the QUICK +# sequence re-captures after the build): a fresh mktemp would orphan the +# first directory, and the exit trap only scrubs the CURRENT run's +# header — an orphaned header is a leaked full-scope token: +RUN=${RUN:-$(mktemp -d)} # private per-run dir for the survival snapshots # Pin the Docker CLI to the SAME daemon socket the supervisor's unit # actually names (its --docker-socket argument, with systemd specifiers # %t = the user runtime dir and %h = the home expanded) — a site-specific @@ -1147,6 +1151,7 @@ if rows: print(s["soulId"], s["intentRevision"], s["launchState"], s.get("containerId") or "-") else: print("")' "$BWS") + NO_BROWSER_SOUL=0 if [ -z "$BS" ]; then # Two cases: step 7 FAILED with a diagnosed stale client (no # managed soul was ever created — export BROWSER_CHECK_FAILED=1 @@ -1154,6 +1159,7 @@ else: # anomaly — refuse): if [ "${BROWSER_CHECK_FAILED:-0}" = "1" ]; then echo "no managed browser soul (stale-client step 7) — restoring settings and removing the workspace" + NO_BROWSER_SOUL=1 else { echo "browser-check soul disappeared from the registry before cleanup — investigate before deleting $BWS (or export BROWSER_CHECK_FAILED=1 if step 7 failed with a stale client)"; exit 1; } fi @@ -1162,11 +1168,12 @@ else: # In recovery (no records tie BWS to this run), confirm the # browser soul separately — mixing two runs' empty probe dirs # must not stop another run's soul unconfirmed: - if [ "${RECOVERY_MINTED:-0}" = "1" ]; then + if [ "${RECOVERY_MINTED:-0}" = "1" ] && [ "${NO_BROWSER_SOUL:-0}" != 1 ]; then echo "Recovery matched browser soul: $BS_ID state=$BS_STATE workspace=$BWS" read -r -p "Stop and delete this BROWSER probe soul too? Type 'yes' to confirm: " BCONFIRM [ "$BCONFIRM" = "yes" ] || { echo "browser probe left running — resolve $BWS manually"; exit 1; } fi + if [ "$NO_BROWSER_SOUL" != 1 ]; then # An EXITED incarnation can still be desired-running (eligible for # recovery) — only a committed stop intent makes it safe to delete # the workspace, so require desiredState=="stopped" too: @@ -1186,6 +1193,7 @@ o=json.load(sys.stdin) assert o.get("outcome")=="verified_empty", o print("stopped browser soul:", o["outcome"])' fi + fi fi # Restore OpenCode's default cwd that the directory step overwrote # (only when the browser check ran — $BWS set). The picker's settings From 18dfce04f1f4ca4dfccf5a41ab9cf509a61ab9e9 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 17:59:31 -0700 Subject: [PATCH 60/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2058?= =?UTF-8?q?=20=E2=80=94=20browser=20check=20matches=20a=20NEW=20soul=20(pr?= =?UTF-8?q?e-creation=20snapshot=20delta)=20and=20restores=20the=20OpenCod?= =?UTF-8?q?e=20default=20immediately=20after=20the=20picker=20save=20lands?= =?UTF-8?q?=20(confound=20window=20shrunk=20to=20seconds);=20writable-root?= =?UTF-8?q?=20retry=20in=20the=20allowed=20fallback;=20CUR=20guarded=20und?= =?UTF-8?q?er=20set=20-u;=20cleanup=20wait=20framed=20as=20belt-and-braces?= =?UTF-8?q?=20with=20its=20residual=20stated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/development/self-hosted-launch-runbook.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 5c5fc2295..0fa4d8fc6 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -1196,11 +1196,13 @@ print("stopped browser soul:", o["outcome"])' fi fi # Restore OpenCode's default cwd that the directory step overwrote - # (only when the browser check ran — $BWS set). The picker's settings - # save is an UNAWAITED async request — first WAIT (bounded) for it to - # land (the visible default becomes $BWS), or the delayed save could - # arrive AFTER this restore and re-point the default at the - # about-to-be-deleted workspace: + # (only when the browser check ran — $BWS set). Step 7 already + # restored it once the picker's async save landed; this is the + # belt-and-braces pass — a save that somehow never landed (or landed + # after step 7's restore window) still gets corrected here before + # the workspace goes away. The wait is bounded; a save arriving after + # BOTH this pass and the post-deletion sweep is the documented + # residual (Settings shows it): if [ -n "${BWS:-}" ]; then for i in $(seq 1 15); do SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ From 591038cf0ef3aa95d53c52870d5e035421d60343 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:01:16 -0700 Subject: [PATCH 61/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2058?= =?UTF-8?q?=20substantive=20=E2=80=94=20browser=20check=20matches=20a=20NE?= =?UTF-8?q?W=20soul=20via=20pre-creation=20snapshot=20delta=20and=20restor?= =?UTF-8?q?es=20the=20OpenCode=20default=20as=20soon=20as=20the=20picker?= =?UTF-8?q?=20save=20lands=20(confound=20window=20seconds,=20not=20minutes?= =?UTF-8?q?);=20allowed=20fallback=20retries=20until=20a=20writable=20root?= =?UTF-8?q?;=20CUR=20guarded=20when=20every=20settings=20read=20fails?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 78 +++++++++++++++---- 1 file changed, 65 insertions(+), 13 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 0fa4d8fc6..fc0ea62fe 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -831,32 +831,82 @@ for r in (afp or []): fi if [ -n "$FIRST_ALLOWED" ]; then # Keep PROBE_ROOT (and the REST probe's WS) where it is — mint a - # SEPARATE browser root under the allowed root and record it: - BROWSER_ROOT=$(mktemp -d "$FIRST_ALLOWED/freshell-probe.XXXXXX") - chmod 0770 "$BROWSER_ROOT" - printf 'BROWSER_ROOT=%q\n' "$BROWSER_ROOT" >> "$D/run.env" - rm -rf "$BWS" - BWS=$(mktemp -d "$BROWSER_ROOT/bws.XXXXXX") - chmod 0770 "$BWS" - printf 'BWS=%q\n' "$BWS" >> "$D/run.env" + # SEPARATE browser root under the first WRITABLE allowed root (a + # valid list may start with read-only roots): + BROWSER_ROOT="" + while read -r A; do + A=${A/#\~/$HOME} + case "$A" in + /*) if BROWSER_ROOT=$(mktemp -d "$A/freshell-probe.XXXXXX" 2>/dev/null); then break; fi ;; + esac + done <<<"$ALLOWED" + if [ -n "$BROWSER_ROOT" ]; then + chmod 0770 "$BROWSER_ROOT" + printf 'BROWSER_ROOT=%q\n' "$BROWSER_ROOT" >> "$D/run.env" + rm -rf "$BWS" + BWS=$(mktemp -d "$BROWSER_ROOT/bws.XXXXXX") + chmod 0770 "$BWS" + printf 'BWS=%q\n' "$BWS" >> "$D/run.env" + fi fi fi # Then WAIT (bounded): # the confirmation dispatches a UI update, terminal.create goes out # from a React effect, and the managed launch is asynchronous — so # the soul may not exist the instant you confirm. Poll up to ~120s: + # Snapshot the soul list BEFORE the pane creation — the check must + # match a NEW soul (a delta), not any pre-existing one: + SOULS_BEFORE=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" \ + http://127.0.0.1:3001/api/runtime/souls \ + | python3 -c 'import json,sys +print(" ".join(x["soulId"] for x in json.load(sys.stdin)["souls"]))') # The manual UI step must finish BEFORE the clock starts — create # the pane, then press ENTER to start the bounded poll: read -r -p "Create the OpenCode pane at $BWS in the hard-refreshed client, then press ENTER to poll: " _ + # Restore the OpenCode default IMMEDIATELY once the picker's async + # save lands, so other clients cannot inherit BWS as their default + # during the poll — the confounding window shrinks from minutes to + # seconds, and cleanup's restore becomes belt-and-braces: + CUR="" + for i in $(seq 1 15); do + SETTINGS_CODE=$(curl -s --connect-timeout 5 --max-time 10 \ + -o "$D/settings.json" -w '%{http_code}' -H @"$D/hdr" \ + http://127.0.0.1:3001/api/settings) + if grep -q '^2' <<<"$SETTINGS_CODE"; then + CUR=$(python3 -c 'import json,sys +d=json.load(open(sys.argv[1])) +assert isinstance(d, dict), "malformed settings response" +d=d.get("data",d) +cc=d.get("codingCli") +assert isinstance(cc, dict), "malformed settings response: no codingCli" +cur=(cc.get("providers") or {}).get("opencode",{}).get("cwd") +print("" if cur is None else cur)' "$D/settings.json" 2>/dev/null) || { rm -f "$D/settings.json"; sleep 2; continue; } + rm -f "$D/settings.json" + if [ "$CUR" = "$BWS" ]; then break; fi + fi + sleep 2 + done + if [ "${CUR:-}" = "$BWS" ] && [ -f "$D/old-opencode-cwd.txt" ]; then + OLD=$(cat "$D/old-opencode-cwd.txt") + BODY=$(python3 -c 'import json,sys +old=sys.argv[1] or None +print(json.dumps({"codingCli":{"providers":{"opencode":{"cwd":old}}}}))' "$OLD") + CODE=$(curl -s --connect-timeout 5 --max-time 30 -o "$D/settings-restore.out" -w '%{http_code}' \ + -X PATCH -H @"$D/hdr" -H "Content-Type: application/json" -d "$BODY" \ + http://127.0.0.1:3001/api/settings) + grep -q '^2' <<<"$CODE" || echo "early default restore failed (HTTP $CODE) — cleanup will retry" + rm -f "$D/settings-restore.out" + fi POLL_DEADLINE=$((SECONDS + 120)) while [ "$SECONDS" -lt "$POLL_DEADLINE" ]; do if curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys +before=set(sys.argv[2].split()) souls=json.load(sys.stdin)["souls"] -rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] +rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1] and s["soulId"] not in before] sids={r["soulId"] for r in rows} assert len(sids)<=1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) -sys.exit(0 if len(sids)==1 and rows[-1].get("launchState")=="running" else 1)' "$BWS"; then +sys.exit(0 if len(sids)==1 and rows[-1].get("launchState")=="running" else 1)' "$BWS" "$SOULS_BEFORE"; then echo "UI pane is managed (after $((SECONDS - POLL_DEADLINE + 120))s)" break fi @@ -866,13 +916,14 @@ sys.exit(0 if len(sids)==1 and rows[-1].get("launchState")=="running" else 1)' " # --max-time flags are what make the poll actually bounded): curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys +before=set(sys.argv[2].split()) souls=json.load(sys.stdin)["souls"] -rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1]] +rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1] and s["soulId"] not in before] sids={r["soulId"] for r in rows} assert len(sids)<=1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) assert rows and rows[-1].get("launchState")=="running", \ - "no running managed soul for the UI pane after ~120s — client stale or not negotiating" -print("UI pane is managed:", rows[-1]["soulId"])' "$BWS" + "no NEW running managed soul for the UI pane after ~120s — client stale or not negotiating" +print("UI pane is managed:", rows[-1]["soulId"])' "$BWS" "$SOULS_BEFORE" ``` If the assertion fails, the client is stale or not negotiating @@ -1221,6 +1272,7 @@ print("" if cur is None else cur)' "$D/settings.json" 2>/dev/null) || { rm -f "$ if [ "$CUR" = "$BWS" ]; then break; fi sleep 2 done + CUR=${CUR:-} # every read failing must not trip set -u downstream # Branch on what the default settled to — cleanup is NEVER wedged: # $BWS -> the picker's save landed; the restore below puts # back the recorded default. From 381c669041aafefdfaff70d79648133f4ae5744f Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:10:18 -0700 Subject: [PATCH 62/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2059?= =?UTF-8?q?=20=E2=80=94=20snapshot=20precedes=20the=20UI=20step;=20Windows?= =?UTF-8?q?-root=20manual=20path=20feeds=20the=20creation=20loop;=20strict?= =?UTF-8?q?-sequence=20capture=20sits=20immediately=20before=20the=20stop;?= =?UTF-8?q?=20browser-probe=20residual=20identity=20risk=20stated=20(no=20?= =?UTF-8?q?UI-exposed=20terminal=20IDs)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 26 +++++++++++++++---- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index fc0ea62fe..20201c3ff 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -233,10 +233,11 @@ the procedure mid-way and leave the service down. Two sequences: immediately before the restart (souls created during the build are otherwise unbaselined and their loss goes undetected), then restart. The short skew window is accepted. -- Strict (changes that must never skew): FIRST re-run the baseline - capture block above (nothing new can be created after the stop, so - a pre-stop baseline is complete), then `systemctl --user stop - freshell-supervisor.service` (the server stops with it via the +- Strict (changes that must never skew): re-run the baseline capture + block IMMEDIATELY before the stop command below (a soul created + between an earlier capture and the stop would be unbaselined; the + next command being the stop makes that window effectively nil), then + `systemctl --user stop freshell-supervisor.service` (the server stops with it via the `Requires=` coupling), THEN run the build command from the Build surface above — never before the stop, or the running supervisor picks up the new session host mid-flight (the exact skew this @@ -828,6 +829,9 @@ for r in (afp or []): /*) ;; *) echo "not a Linux absolute path — aborting"; exit 1 ;; esac + # Feed the typed path into the creation loop below (it iterates + # over $ALLOWED): + ALLOWED="$FIRST_ALLOWED" fi if [ -n "$FIRST_ALLOWED" ]; then # Keep PROBE_ROOT (and the REST probe's WS) where it is — mint a @@ -855,7 +859,10 @@ for r in (afp or []): # from a React effect, and the managed launch is asynchronous — so # the soul may not exist the instant you confirm. Poll up to ~120s: # Snapshot the soul list BEFORE the pane creation — the check must - # match a NEW soul (a delta), not any pre-existing one: + # match a NEW soul (a delta), not any pre-existing one. The snapshot + # MUST precede the UI step below, or the probe's own soul lands in + # the baseline and the delta assertion rejects a correctly managed + # pane: SOULS_BEFORE=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" \ http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys @@ -929,6 +936,15 @@ print("UI pane is managed:", rows[-1]["soulId"])' "$BWS" "$SOULS_BEFORE" If the assertion fails, the client is stale or not negotiating `managedRuntimeV1` — rebuild/redeploy the client and hard-refresh again. Close the test pane. + RESIDUAL RISK (documented, not eliminated): the UI does not expose + the tested pane's terminal ID, so the check identifies the pane by + workspace + creation-delta, not by exact terminal identity. A + concurrent client that creates an OpenCode pane in $BWS inside the + seconds between this client's picker save and the immediate default + restore could still confound the match. Run this check when no + other client is likely to create OpenCode panes (the single-operator + self-hosted norm), and treat any surprise match with suspicion — + the cleanup confirmations still gate destructive stops. 8. Probe cleanup (LAST): probe souls are NOT auto-collected — the server's idle reaper explicitly skips managed terminals (their lifetime is supervisor-owned) and the supervisor has no idle reaper, From 45510fe8dbfd487550e93af6cb1dd0597db91ab4 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:19:13 -0700 Subject: [PATCH 63/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2060?= =?UTF-8?q?=20=E2=80=94=20snapshot-before-confirm=20enforced=20in=20the=20?= =?UTF-8?q?prose;=20BWS=20matched=20by=20its=20canonical=20form=20(the=20s?= =?UTF-8?q?erver=20canonicalizes=20terminalCwd;=20symlinked=20allowed=20ro?= =?UTF-8?q?ots=20match),=20BWS=5FMATCH=20recorded=20and=20restored?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- .../development/self-hosted-launch-runbook.md | 26 ++++++++++++++----- 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index 20201c3ff..ef9dcbb6d 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -739,7 +739,11 @@ print(next(s["soulId"] for s in souls if s.get("terminalId")==sys.argv[1]))' "$T fi BWS=$(mktemp -d "$PROBE_ROOT/bws.XXXXXX") # unique per run chmod 0770 "$BWS" # same in-soul-shell access requirement as $WS - printf 'BWS=%q\n' "$BWS" >> "$D/run.env" + # The server CANONICALIZES the cwd when it stores terminalCwd — a + # symlinked parent in an allowed root means the raw $BWS spelling + # never matches the soul record. Match on the canonical form: + BWS_MATCH=$(realpath -e "$BWS") + printf 'BWS=%q\nBWS_MATCH=%q\n' "$BWS" "$BWS_MATCH" >> "$D/run.env" # The directory picker PERSISTS the chosen path as OpenCode's default # cwd (the confirm handler patches settings) — record the current # default (VERIFIED read: an error response must not masquerade as @@ -765,8 +769,11 @@ print("" if cur is None else cur)' "$D/settings.json" > "$D/old-opencode-cwd.txt # correctly deployed client). A plain shell pane will NOT do either: # shell panes skip the directory step and always take the default # workspace. OpenCode's directory step has a free-text PATH INPUT: - # type the absolute $BWS path there and confirm. If the picker - # REJECTS the path (the directory is validated against the + # type the absolute $BWS path there — but DO NOT CONFIRM YET: the + # block below snapshots the soul list FIRST, and its prompt tells + # you when to confirm (a pane created before the snapshot lands in + # the baseline and the delta assertion would reject a correct + # deploy). If the picker REJECTS the path (the directory is validated against the # `allowedFilePaths` setting — a deployment whose allowed roots # exclude $HOME will 403 the probe root), mint the probe root under # an allowed root instead and record it so the cleanup guard @@ -913,7 +920,7 @@ souls=json.load(sys.stdin)["souls"] rows=[s for s in souls if s.get("terminalCwd")==sys.argv[1] and s["soulId"] not in before] sids={r["soulId"] for r in rows} assert len(sids)<=1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) -sys.exit(0 if len(sids)==1 and rows[-1].get("launchState")=="running" else 1)' "$BWS" "$SOULS_BEFORE"; then +sys.exit(0 if len(sids)==1 and rows[-1].get("launchState")=="running" else 1)' "$BWS_MATCH" "$SOULS_BEFORE"; then echo "UI pane is managed (after $((SECONDS - POLL_DEADLINE + 120))s)" break fi @@ -930,7 +937,7 @@ sids={r["soulId"] for r in rows} assert len(sids)<=1, ("workspace ambiguous (mint a fresh BWS and retry)", rows) assert rows and rows[-1].get("launchState")=="running", \ "no NEW running managed soul for the UI pane after ~120s — client stale or not negotiating" -print("UI pane is managed:", rows[-1]["soulId"])' "$BWS" "$SOULS_BEFORE" +print("UI pane is managed:", rows[-1]["soulId"])' "$BWS_MATCH" "$SOULS_BEFORE" ``` If the assertion fails, the client is stale or not negotiating @@ -1148,6 +1155,10 @@ PY echo "REFUSING: D=$D records browser workspace $RECORDED_BWS, but BWS=$BWS — mixed runs; resolve manually" exit 1 fi + RECORDED_BWS_MATCH=$(bash -c '. "$1"; printf "%s" "${BWS_MATCH:-}"' _ "$D/run.env") + if [ -n "$RECORDED_BWS_MATCH" ]; then + BWS_MATCH="$RECORDED_BWS_MATCH" + fi fi fi fi @@ -1204,6 +1215,9 @@ print("stopped:", o["outcome"])' # Look it up by workspace in ANY launchState — a blocked or exited # probe soul still must not be orphaned against a deleted workspace: if [ -n "${BWS:-}" ]; then + # Match the server's canonical spelling (BWS_MATCH from the + # records; realpath of $BWS in recovery): + BWS_LOOKUP="${BWS_MATCH:-$(realpath -e "$BWS" 2>/dev/null || printf '%s' "$BWS")}" BS=$(curl -s --connect-timeout 5 --max-time 10 -H @"$D/hdr" http://127.0.0.1:3001/api/runtime/souls \ | python3 -c 'import json,sys souls=json.load(sys.stdin)["souls"] @@ -1217,7 +1231,7 @@ if rows: s=rows[-1] print(s["soulId"], s["intentRevision"], s["launchState"], s.get("containerId") or "-") else: - print("")' "$BWS") + print("")' "$BWS_LOOKUP") NO_BROWSER_SOUL=0 if [ -z "$BS" ]; then # Two cases: step 7 FAILED with a diagnosed stale client (no From a73d3618188e492074a3d35f64b64df4506ad8a6 Mon Sep 17 00:00:00 2001 From: Dan Shapiro <3732858+danshapiro@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:31:20 -0700 Subject: [PATCH 64/64] =?UTF-8?q?docs(runbook):=20fresheyes=20round=2061?= =?UTF-8?q?=20(final)=20=E2=80=94=20allowed-roots=20fallback=20recomputes?= =?UTF-8?q?=20BWS=5FMATCH=20for=20the=20new=20workspace?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/development/self-hosted-launch-runbook.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/development/self-hosted-launch-runbook.md b/docs/development/self-hosted-launch-runbook.md index ef9dcbb6d..be3a62de2 100644 --- a/docs/development/self-hosted-launch-runbook.md +++ b/docs/development/self-hosted-launch-runbook.md @@ -857,7 +857,10 @@ for r in (afp or []): rm -rf "$BWS" BWS=$(mktemp -d "$BROWSER_ROOT/bws.XXXXXX") chmod 0770 "$BWS" - printf 'BWS=%q\n' "$BWS" >> "$D/run.env" + # Recompute the canonical match path for the NEW workspace — a + # stale BWS_MATCH would search for the deleted old one: + BWS_MATCH=$(realpath -e "$BWS") + printf 'BWS=%q\nBWS_MATCH=%q\n' "$BWS" "$BWS_MATCH" >> "$D/run.env" fi fi fi