feat(paseo): add backup/restore scripts to preserve state across sandbox recreation

integrations/isolation/acq-kits/paseo/README.md

@@ -141,6 +141,63 @@ To override the location, pre-set `worktrees.root` to an absolute path in
 replaced on the next run — set it via the entrypoint's working directory instead,
 or remove the shim's pinning if you need a custom fixed root.
 
+## Backing up and restoring Paseo state across recreation
+
+An `acq` sandbox is ephemeral: only your host bind mounts survive its
+destruction. The whole guest home — including `$PASEO_HOME` (default `~/.paseo`:
+`config.json`, the projects registry, and Paseo's session/agent records) — is
+**lost** when you `acq rm` and recreate. Your git worktrees themselves survive
+(they live under `<project>/.paseo-worktrees` on the host mount), but after a
+recreate Paseo no longer has their records, so they show up orphaned.
+
+Two host-side helper scripts snapshot Paseo state before teardown and replay it
+into the new sandbox afterward:
+
+```bash
+# Before recreating — snapshot state to the host (default ~/.acq-paseo-backups).
+# Dry-run first (prints what it would do), then --apply to actually write it:
+scripts/paseo-backup <sandbox>
+scripts/paseo-backup --apply <sandbox>
+
+# Recreate the sandbox as usual:
+acq rm <sandbox>
+acq run opencode /path/to/your/project        # (or acq create …)
+
+# After the new sandbox is up — restore. With no tar given it auto-selects the
+# most recent backup for that sandbox. Dry-run first, then --apply:
+scripts/paseo-restore <sandbox>
+scripts/paseo-restore --apply <sandbox>
+```
+
+`paseo-restore` prunes legacy runtime logs from older backups, streams the
+snapshot back into a temporary directory in the sandbox, atomically swaps it into
+`$PASEO_HOME` after a complete extract, and restarts the daemon so it re-reads
+the restored config and re-recognizes your projects and worktrees.
+
+- **What's captured:** the `$PASEO_HOME` tree (`config.json`, `projects/`,
+  `schedules/`, session records) minus files that should not travel between
+  sandboxes: `paseo.pid` (a PID lock the daemon recreates), `daemon.log`,
+  `daemon.log.*`, `daemon.log.txt`, and `*-daemon.log` (live or rotated runtime
+  logs — excluded so concurrent writes cannot fail the archive and restore does
+  not crawl through non-state data), the per-install identity `daemon-keypair.json`
+  / `cli-client-id` / `server-id` (regenerated on boot), and `models/` (large,
+  re-downloadable model caches — a speech model alone is ~460 MB). Logs under
+  `~/.local/state/paseo` are also regenerated every boot and are not captured.
+- **Worktrees:** their contents are **not** in the backup — they already persist
+  on the host mount. The restored `$PASEO_HOME` is what makes Paseo re-recognize
+  them, as long as the project is re-mounted at the **same guest path** (acq's
+  normal behavior).
+- **`worktrees.root`:** a restored `config.json` carries the old sandbox's root.
+  It self-corrects on the next `acq run` (the wrapper re-pins it). If you
+  restored into a **detached** `acq create`, run `acq run opencode <project>`
+  once so the root matches this sandbox's mount.
+- **Safety:** both scripts default to dry-run; pass `--apply` to actually write
+  or restore. The backup is read-only against the sandbox.
+
+Backups are plain tars — inspect one with `tar -tf <file>`. See
+`docs/decisions/backup-restore-paseo-state.md` for the design and the worktree
+rationale.
+
 ## No shared session with a terminal TUI
 
 Unlike a shared-server setup (e.g. the `openchamber` kit), Paseo is **not** an
@@ -239,6 +296,8 @@ paseo/
 ├── README.md                       # this file
 ├── TROUBLESHOOTING.md              # failure modes and fixes
 ├── scripts/verify                  # host-side live check
+├── scripts/paseo-backup            # snapshot $PASEO_HOME to a host tar (dry-run default)
+├── scripts/paseo-restore           # replay a snapshot into a new sandbox + bounce the daemon
 └── docs/decisions/                 # design records (ADRs)
 ```
 
@@ -248,5 +307,7 @@ Rationale and the decisions behind this kit's structure live in
 [`docs/decisions/`](docs/decisions/) — notably why one daemon serves everything
 on a single port, why worktrees can only be pinned to the first project dir (and
 why that needs a daemon restart), why the startup script owns the daemon while
-the entrypoint pins worktrees, why the install runs at startup, and why the
-entrypoint is split into a thin agent-named wrapper over a generic kit shim.
+the entrypoint pins worktrees, why the install runs at startup, why the
+entrypoint is split into a thin agent-named wrapper over a generic kit shim, and
+why Paseo state is preserved across sandbox recreation with backup/restore
+scripts rather than a persistent volume.

integrations/isolation/acq-kits/paseo/TROUBLESHOOTING.md

@@ -328,6 +328,55 @@ rewrites `config.json` when the value differs. If you need a custom fixed root,
 don't rely on the wrapper's pin — the current kit does not expose an override env
 var.
 
+## I recreated the sandbox and lost my Paseo state / my worktrees show as orphaned
+
+**Symptom.** After `acq rm` + a fresh `acq create`/`acq run`, the Paseo UI has no
+projects, no session history, and any git worktrees under
+`<project>/.paseo-worktrees` are present on disk but unknown to Paseo.
+
+**Cause.** The sandbox is ephemeral — only host bind mounts survive. `$PASEO_HOME`
+(default `~/.paseo`: `config.json`, the projects registry, session/agent records)
+is lost on recreate. The worktree *files* persist (they live on the host mount),
+but Paseo's *records* of them do not, so they look orphaned.
+
+**Prevention + recovery.** Use the kit's host-side scripts. Snapshot before you
+recreate, restore after the new sandbox is up:
+
+```bash
+# BEFORE recreating (dry-run prints what it would do; --apply writes it):
+scripts/paseo-backup --apply <sandbox>      # → ~/.acq-paseo-backups/paseo-state-<sandbox>-<UTC>.tar
+
+# recreate as usual …
+acq rm <sandbox>
+acq run opencode /path/to/your/project
+
+# AFTER the new sandbox is up (auto-selects the most recent backup for it):
+scripts/paseo-restore --apply <sandbox>
+```
+
+`paseo-restore` streams the snapshot back into `$PASEO_HOME` and bounces the
+daemon so it re-reads the restored config and re-recognizes your projects and
+worktrees. Confirm it took:
+
+```bash
+acq exec <sandbox> -- sh -c 'cat "${PASEO_HOME:-$HOME/.paseo}/config.json"; ls "${PASEO_HOME:-$HOME/.paseo}/projects" 2>/dev/null'
+acq exec <sandbox> -- sh -c 'curl -fsS http://127.0.0.1:6767/api/health && echo OK'
+```
+
+Notes:
+
+- **You must have made a backup first** — restore replays a prior snapshot; it
+  cannot recover state that was never captured. Recreation is unplanned by
+  definition, so back up proactively when your Paseo state matters.
+- **`worktrees.root` self-corrects on the next `acq run`** (the wrapper re-pins
+  it). If you restored into a **detached** `acq create`, run
+  `acq run opencode <project>` once so the root matches this sandbox's mount.
+- **Same guest path required.** Worktrees line up with the restored records only
+  if the project is re-mounted at the same guest path (acq's normal behavior).
+- **`paseo.pid` is intentionally not in the backup**; the daemon recreates it.
+
+See `docs/decisions/backup-restore-paseo-state.md` for the design.
+
 ## Endless `relay_error` / `relay_control_disconnected` in the daemon log
 
 **Symptom.** The daemon log fills with, every ~30s:

integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md

@@ -0,0 +1,208 @@
+# Decision: back up and restore Paseo state across sandbox recreation
+
+**Status:** accepted
+
+## Context
+
+An `acq` sandbox is an ephemeral container. Only the host bind mounts survive its
+destruction; the entire guest filesystem — including `$PASEO_HOME` (default
+`~/.paseo`: `config.json`, the `projects/projects.json` registry, and Paseo's
+session/agent records) and `~/.local/state/paseo` (logs) — is lost on
+`acq rm` + a fresh create. So every unplanned sandbox recreation drops the user's
+Paseo state, and the on-disk git worktrees that DO persist (see below) show up
+"orphaned" in the UI because Paseo no longer holds their records.
+
+We evaluated three durable-storage approaches before this one:
+
+1. **Relocate `$PASEO_HOME` onto the first project mount.** Survives recreate
+   today, but couples Paseo's home to a repo directory (needs `.gitignore`
+   hygiene) and scopes state to one project.
+2. **A dedicated host state mount** (`PASEO_HOME` on an extra mounted dir).
+   Clean, but forces an extra mount argument on every `acq run`/`acq create`.
+3. **A kit-declared persistent volume.** The `hybrid/v1` schema already has a
+   `volumes` block, but its documented semantics are the opposite of what is
+   needed here: it is **creation-time only, mounts UNSEEDED, and is per-sandbox
+   scratch storage** — it does not outlive the sandbox, and the translate layer
+   that maps it lives in the separate `acq`/quickstart repo, not this one. It
+   therefore cannot provide cross-recreate persistence without an upstream
+   `acq`-side change (see the `volumes` amendment in
+   `../../../docs/decisions/0001-neutral-hybrid-v1-acq-kits.md`).
+
+## Decision
+
+**Ship two host-side helper scripts — `scripts/paseo-backup` and
+`scripts/paseo-restore` — that snapshot `$PASEO_HOME` to a host tar before
+teardown and replay it into a new sandbox afterward.** They use only `acq exec`
+(already the kit's universal in-guest mechanism); no new `acq` capability, no
+persistent-storage primitive, no coupling of Paseo state to a project repo, and
+no extra mount argument.
+
+- **`paseo-backup [--apply] <sandbox> [dest-dir]`** tars
+  `${PASEO_HOME:-$HOME/.paseo}` (minus the exclusions below) inside the guest and
+  emits it base64-encoded over `acq exec` stdout; the host decodes it into
+  `<dest>/paseo-state-<sandbox>-<UTC>.tar` (default dest `~/.acq-paseo-backups`).
+  It writes to a temp file and atomically renames on success, and verifies the
+  decoded result is a valid tar before promoting it — an interrupted capture
+  never leaves a truncated archive.
+- **`paseo-restore [--apply] <sandbox> [backup-tar]`** prunes legacy runtime logs
+  from older backups, base64-encodes the backup tar on the host, pipes the text
+  into the guest which decodes and extracts it into a temporary sibling directory,
+  atomically swaps that directory into `${PASEO_HOME:-$HOME/.paseo}`, then bounces
+  the daemon so it re-reads the restored config/projects. With no `backup-tar` it
+  auto-selects the most recent backup for that sandbox.
+- Both **default to dry-run**; `--apply` is required to write or mutate anything.
+
+## Transport is base64 text, not raw binary
+
+`acq` must work over **both** isolation backends (msb and sbx). Streaming a raw
+binary tar over `acq exec` stdin/stdout was observed to **hang** on msb: a guest
+`cat | wc -c` fed the tar bytes never saw EOF, so `tar -xf -` blocked forever.
+Plain multi-line **text** stdin, by contrast, reaches EOF normally (verified).
+`msb exec` exposes a `--stream` flag for byte-faithful binary I/O, but relying on
+it would be backend-specific.
+
+So the archive is transported as **base64 text** in both directions — the guest
+encodes on backup, the host encodes on restore — keeping the payload in the safe
+text channel that behaves identically on every backend. `base64` is required on
+both the host and in the guest image (present on the opencode base image). The
+cost is ~33% transport inflation, which is immaterial once the snapshot is kept
+small (see exclusions). This is why excluding the model cache matters: base64 of
+a ~460MB model cache is ~640MB of text piped through `acq exec`, which crawled to
+an apparent hang — trimming the snapshot to real state keeps the text channel
+fast.
+
+## Capture is race-tolerant: snapshot-in-guest, then classify the tar exit code
+
+`acq exec` **faithfully propagates the guest command's exit code** (verified:
+`exit 7` in the guest surfaces as host exit 7). That is a feature — it lets the
+scripts detect real failures — but it means a naive `tar -cf - .` fails
+on an *expected, non-fatal* condition: the daemon writes under `$PASEO_HOME`
+while tar reads it, and GNU tar exits **1** ("file changed as we read it"), or
+**2** when a whole entry is affected, *while still producing a valid archive of
+everything else*. The first live `--apply` failed with exactly this (`rc=2`).
+
+The capture therefore does two things in the guest:
+
+1. **Snapshots to a guest temp file first**, then `cat`s that settled file to
+   stdout — so the bytes streamed to the host can't race the daemon.
+2. **Classifies tar's exit code**: `0` = clean; `1`/`2` = files
+   changed/vanished mid-read (expected here — the archive is usable); `>= 3` =
+   a genuine failure (missing dir, write error), which aborts. `daemon.log` (the
+   busiest file) is excluded outright, and `--warning=no-file-changed
+   --ignore-failed-read` quiet the remaining churn.
+
+Because the guest capture now exits non-zero **only** on a real failure, the host
+side *can* trust the propagated exit code, and additionally verifies the result
+is a non-empty, valid tar (`tar -tf`) before atomically renaming it into place.
+`paseo-restore` likewise checks both the propagated exit code and a completion
+marker the guest prints, so a partial extract is never mistaken for success.
+
+## What the snapshot includes and excludes
+
+- **Included:** the `$PASEO_HOME` tree (`config.json`, `projects/`, `agents/`,
+  session records, anything else Paseo persists there) minus the specific
+  exclusions below. Capturing the tree broadly — rather than an allowlist of
+  known files — means a future Paseo state file is preserved automatically.
+- **Excluded — `paseo.pid`:** a supervisor PID lock. Replaying a stale PID into a
+  fresh container is misleading; the daemon recreates it on boot. (Paseo's lock is
+  stale-tolerant, so a leftover would be reclaimed anyway, but there is no reason
+  to carry it.)
+- **Excluded — `daemon.log`, `daemon.log.*`, `daemon.log.txt`, and
+  `*-daemon.log`:** live or rotated daemon logs can be multi-MB and are not
+  restorable state. Because live logs are appended *while the tar reads them*, GNU
+  tar reports "file changed as we read it" and exits non-zero (rc 1, or rc 2 when
+  a whole entry is affected) — the observed cause of the first live backup
+  failing. Rotated logs have also made restore appear hung because they can be
+  streamed before useful state files over acq's base64 text channel.
+- **Excluded — `daemon-keypair.json`, `cli-client-id`, `server-id`:**
+  per-daemon/-install identity that the daemon regenerates on first boot.
+  Carrying it into a *different* sandbox is pointless at best and confusing at
+  worst.
+- **Excluded — `models/`:** downloaded model caches (e.g. a ~460 MB sherpa-onnx
+  speech model observed on a live sandbox). Large, re-downloadable, and not
+  state; including it both bloats the archive and makes the base64 transport
+  crawl. The daemon re-fetches on demand.
+- **Not captured — `~/.local/state/paseo`:** logs, install log, CA bundle. All
+  regenerated every boot.
+- **Not captured — git worktree contents.** See below.
+
+## Why worktrees are left on the project mount, not moved into the backup
+
+The kit pins Paseo's `worktrees.root` to `<primary-project>/.paseo-worktrees`,
+i.e. **inside a host bind mount** (see `worktrees-root-global-only.md`). A git
+worktree stores absolute paths in both link halves (the worktree's `.git` file
+points at `<mainrepo>/.git/worktrees/<name>`, which points back at the worktree's
+absolute path). Because both the worktrees and their main repos live on the same
+host mount, and `acq` re-mounts a project at the **same guest path** on recreate,
+the worktree files and their git linkage **already survive recreation on their
+own**. Moving worktrees into `$PASEO_HOME` so they would ride the backup tar would
+break that linkage (the main repo would not move with them) and would make them
+inaccessible from the host while active.
+
+So the real gap on recreate is not the worktree files — it is Paseo's *knowledge*
+of them (its session/worktree records under `$PASEO_HOME`). Restoring the
+`$PASEO_HOME` snapshot is exactly what makes Paseo re-recognize the intact on-disk
+worktrees. This keeps the design simple and avoids leaving either the host or
+Paseo in an inconsistent state.
+
+## The restore daemon bounce must clear the stale PID lock
+
+After extracting the snapshot, `paseo-restore` restarts the daemon so it re-reads
+the restored `config.json` and projects. The kit runs the daemon under a
+`supervisor:paseo-daemon` respawn loop, so the bounce **kills the daemon child**
+(never the supervisor) and lets the supervisor relaunch it.
+
+But killing the daemon leaves its `$PASEO_HOME/paseo.pid` lock behind, and
+Paseo only treats that lock as stale after **five minutes**. The supervisor
+respawns every ~5 s, so without intervention every relaunch loses the race —
+`Failed to acquire PID lock due to race condition` on repeat, and the daemon
+never comes back (observed live). The bounce therefore **kills the daemon, waits
+for it to actually exit, then removes `paseo.pid`** so the next respawn acquires
+the lock cleanly. The lock is meaningless once its owner is dead, so removing it
+is safe. (This is also why `paseo.pid` is excluded from the snapshot — a restored
+stale lock would reintroduce the same stall.)
+
+## `worktrees.root` reconciliation on restore
+
+A restored `config.json` carries the **old** sandbox's absolute `worktrees.root`.
+On the new sandbox the kit's `opencode` wrapper re-pins `worktrees.root` to
+`<primary-project>/.paseo-worktrees` on the next `acq run` and bounces the daemon
+(see `supervisor-owns-daemon-wrapper-pins-worktrees.md`), so a stale value
+self-corrects. When restoring into a detached `acq create`, the user runs
+`acq run <agent> <project>` once (documented in the script output, README, and
+TROUBLESHOOTING) to trigger the re-pin.
+
+## The filename contract
+
+`paseo-backup` writes `paseo-state-<sandbox>-<UTC>.tar` where `<UTC>` is a
+colon-free ISO-8601 basic timestamp (`YYYYMMDDTHHMMSSZ`) so the name is valid on
+every filesystem **and** sorts lexicographically by time. `paseo-restore`'s
+"most recent backup" default relies on that ordering (lexical sort, last entry,
+filtered to the sandbox prefix). The two scripts must change this format in
+lockstep; both headers state the coupling.
+
+## Consequences
+
+- Recreation becomes a documented three-step ritual: `paseo-backup --apply` →
+  recreate → `paseo-restore --apply` (then one `acq run` to re-pin worktrees).
+- No dependency on unshipped `acq` volume semantics; entirely inside this kit.
+- Backups are plain tars on the host, inspectable with `tar -tf` and portable
+  between sandboxes if desired.
+- Manual by design (per the current scope): there is no automatic
+  backup-on-teardown and no one-shot recreate wrapper. Those were considered and
+  deliberately not built now.
+- Sandbox names flow into a filename glob and an `acq exec` target, so both
+  scripts constrain the name to `[A-Za-z0-9._-]` (SI-10).
+
+## Links
+
+- `worktrees-root-global-only.md` — why worktrees live on the project mount and
+  why `worktrees.root` is re-pinned per run.
+- `supervisor-owns-daemon-wrapper-pins-worktrees.md` — the marker-scoped daemon
+  bounce that `paseo-restore` reuses.
+- `../../../docs/decisions/0001-neutral-hybrid-v1-acq-kits.md` — the `volumes`
+  amendment whose creation-time/unseeded/per-sandbox semantics ruled it out here.
+- `../../files/home/paseo-set-worktrees-root.mjs`,
+  `../../files/home/paseo-agent-shim` — the `${PASEO_HOME:-$HOME/.paseo}`
+  resolution the scripts mirror.
+- `../../scripts/paseo-backup`, `../../scripts/paseo-restore`.