diff --git a/integrations/isolation/acq-kits/paseo/README.md b/integrations/isolation/acq-kits/paseo/README.md index 8c9a2cb7..f973cf79 100644 --- a/integrations/isolation/acq-kits/paseo/README.md +++ b/integrations/isolation/acq-kits/paseo/README.md @@ -141,6 +141,63 @@ To override the location, pre-set `worktrees.root` to an absolute path in replaced on the next run — set it via the entrypoint's working directory instead, or remove the shim's pinning if you need a custom fixed root. +## Backing up and restoring Paseo state across recreation + +An `acq` sandbox is ephemeral: only your host bind mounts survive its +destruction. The whole guest home — including `$PASEO_HOME` (default `~/.paseo`: +`config.json`, the projects registry, and Paseo's session/agent records) — is +**lost** when you `acq rm` and recreate. Your git worktrees themselves survive +(they live under `/.paseo-worktrees` on the host mount), but after a +recreate Paseo no longer has their records, so they show up orphaned. + +Two host-side helper scripts snapshot Paseo state before teardown and replay it +into the new sandbox afterward: + +```bash +# Before recreating — snapshot state to the host (default ~/.acq-paseo-backups). +# Dry-run first (prints what it would do), then --apply to actually write it: +scripts/paseo-backup +scripts/paseo-backup --apply + +# Recreate the sandbox as usual: +acq rm +acq run opencode /path/to/your/project # (or acq create …) + +# After the new sandbox is up — restore. With no tar given it auto-selects the +# most recent backup for that sandbox. Dry-run first, then --apply: +scripts/paseo-restore +scripts/paseo-restore --apply +``` + +`paseo-restore` prunes legacy runtime logs from older backups, streams the +snapshot back into a temporary directory in the sandbox, atomically swaps it into +`$PASEO_HOME` after a complete extract, and restarts the daemon so it re-reads +the restored config and re-recognizes your projects and worktrees. + +- **What's captured:** the `$PASEO_HOME` tree (`config.json`, `projects/`, + `schedules/`, session records) minus files that should not travel between + sandboxes: `paseo.pid` (a PID lock the daemon recreates), `daemon.log`, + `daemon.log.*`, `daemon.log.txt`, and `*-daemon.log` (live or rotated runtime + logs — excluded so concurrent writes cannot fail the archive and restore does + not crawl through non-state data), the per-install identity `daemon-keypair.json` + / `cli-client-id` / `server-id` (regenerated on boot), and `models/` (large, + re-downloadable model caches — a speech model alone is ~460 MB). Logs under + `~/.local/state/paseo` are also regenerated every boot and are not captured. +- **Worktrees:** their contents are **not** in the backup — they already persist + on the host mount. The restored `$PASEO_HOME` is what makes Paseo re-recognize + them, as long as the project is re-mounted at the **same guest path** (acq's + normal behavior). +- **`worktrees.root`:** a restored `config.json` carries the old sandbox's root. + It self-corrects on the next `acq run` (the wrapper re-pins it). If you + restored into a **detached** `acq create`, run `acq run opencode ` + once so the root matches this sandbox's mount. +- **Safety:** both scripts default to dry-run; pass `--apply` to actually write + or restore. The backup is read-only against the sandbox. + +Backups are plain tars — inspect one with `tar -tf `. See +`docs/decisions/backup-restore-paseo-state.md` for the design and the worktree +rationale. + ## No shared session with a terminal TUI Unlike a shared-server setup (e.g. the `openchamber` kit), Paseo is **not** an @@ -239,6 +296,8 @@ paseo/ ├── README.md # this file ├── TROUBLESHOOTING.md # failure modes and fixes ├── scripts/verify # host-side live check +├── scripts/paseo-backup # snapshot $PASEO_HOME to a host tar (dry-run default) +├── scripts/paseo-restore # replay a snapshot into a new sandbox + bounce the daemon └── docs/decisions/ # design records (ADRs) ``` @@ -248,5 +307,7 @@ Rationale and the decisions behind this kit's structure live in [`docs/decisions/`](docs/decisions/) — notably why one daemon serves everything on a single port, why worktrees can only be pinned to the first project dir (and why that needs a daemon restart), why the startup script owns the daemon while -the entrypoint pins worktrees, why the install runs at startup, and why the -entrypoint is split into a thin agent-named wrapper over a generic kit shim. +the entrypoint pins worktrees, why the install runs at startup, why the +entrypoint is split into a thin agent-named wrapper over a generic kit shim, and +why Paseo state is preserved across sandbox recreation with backup/restore +scripts rather than a persistent volume. diff --git a/integrations/isolation/acq-kits/paseo/TROUBLESHOOTING.md b/integrations/isolation/acq-kits/paseo/TROUBLESHOOTING.md index d5a62788..a4b4463f 100644 --- a/integrations/isolation/acq-kits/paseo/TROUBLESHOOTING.md +++ b/integrations/isolation/acq-kits/paseo/TROUBLESHOOTING.md @@ -328,6 +328,55 @@ rewrites `config.json` when the value differs. If you need a custom fixed root, don't rely on the wrapper's pin — the current kit does not expose an override env var. +## I recreated the sandbox and lost my Paseo state / my worktrees show as orphaned + +**Symptom.** After `acq rm` + a fresh `acq create`/`acq run`, the Paseo UI has no +projects, no session history, and any git worktrees under +`/.paseo-worktrees` are present on disk but unknown to Paseo. + +**Cause.** The sandbox is ephemeral — only host bind mounts survive. `$PASEO_HOME` +(default `~/.paseo`: `config.json`, the projects registry, session/agent records) +is lost on recreate. The worktree *files* persist (they live on the host mount), +but Paseo's *records* of them do not, so they look orphaned. + +**Prevention + recovery.** Use the kit's host-side scripts. Snapshot before you +recreate, restore after the new sandbox is up: + +```bash +# BEFORE recreating (dry-run prints what it would do; --apply writes it): +scripts/paseo-backup --apply # → ~/.acq-paseo-backups/paseo-state--.tar + +# recreate as usual … +acq rm +acq run opencode /path/to/your/project + +# AFTER the new sandbox is up (auto-selects the most recent backup for it): +scripts/paseo-restore --apply +``` + +`paseo-restore` streams the snapshot back into `$PASEO_HOME` and bounces the +daemon so it re-reads the restored config and re-recognizes your projects and +worktrees. Confirm it took: + +```bash +acq exec -- sh -c 'cat "${PASEO_HOME:-$HOME/.paseo}/config.json"; ls "${PASEO_HOME:-$HOME/.paseo}/projects" 2>/dev/null' +acq exec -- sh -c 'curl -fsS http://127.0.0.1:6767/api/health && echo OK' +``` + +Notes: + +- **You must have made a backup first** — restore replays a prior snapshot; it + cannot recover state that was never captured. Recreation is unplanned by + definition, so back up proactively when your Paseo state matters. +- **`worktrees.root` self-corrects on the next `acq run`** (the wrapper re-pins + it). If you restored into a **detached** `acq create`, run + `acq run opencode ` once so the root matches this sandbox's mount. +- **Same guest path required.** Worktrees line up with the restored records only + if the project is re-mounted at the same guest path (acq's normal behavior). +- **`paseo.pid` is intentionally not in the backup**; the daemon recreates it. + +See `docs/decisions/backup-restore-paseo-state.md` for the design. + ## Endless `relay_error` / `relay_control_disconnected` in the daemon log **Symptom.** The daemon log fills with, every ~30s: diff --git a/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md b/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md new file mode 100644 index 00000000..ee7d6c13 --- /dev/null +++ b/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md @@ -0,0 +1,208 @@ +# Decision: back up and restore Paseo state across sandbox recreation + +**Status:** accepted + +## Context + +An `acq` sandbox is an ephemeral container. Only the host bind mounts survive its +destruction; the entire guest filesystem — including `$PASEO_HOME` (default +`~/.paseo`: `config.json`, the `projects/projects.json` registry, and Paseo's +session/agent records) and `~/.local/state/paseo` (logs) — is lost on +`acq rm` + a fresh create. So every unplanned sandbox recreation drops the user's +Paseo state, and the on-disk git worktrees that DO persist (see below) show up +"orphaned" in the UI because Paseo no longer holds their records. + +We evaluated three durable-storage approaches before this one: + +1. **Relocate `$PASEO_HOME` onto the first project mount.** Survives recreate + today, but couples Paseo's home to a repo directory (needs `.gitignore` + hygiene) and scopes state to one project. +2. **A dedicated host state mount** (`PASEO_HOME` on an extra mounted dir). + Clean, but forces an extra mount argument on every `acq run`/`acq create`. +3. **A kit-declared persistent volume.** The `hybrid/v1` schema already has a + `volumes` block, but its documented semantics are the opposite of what is + needed here: it is **creation-time only, mounts UNSEEDED, and is per-sandbox + scratch storage** — it does not outlive the sandbox, and the translate layer + that maps it lives in the separate `acq`/quickstart repo, not this one. It + therefore cannot provide cross-recreate persistence without an upstream + `acq`-side change (see the `volumes` amendment in + `../../../docs/decisions/0001-neutral-hybrid-v1-acq-kits.md`). + +## Decision + +**Ship two host-side helper scripts — `scripts/paseo-backup` and +`scripts/paseo-restore` — that snapshot `$PASEO_HOME` to a host tar before +teardown and replay it into a new sandbox afterward.** They use only `acq exec` +(already the kit's universal in-guest mechanism); no new `acq` capability, no +persistent-storage primitive, no coupling of Paseo state to a project repo, and +no extra mount argument. + +- **`paseo-backup [--apply] [dest-dir]`** tars + `${PASEO_HOME:-$HOME/.paseo}` (minus the exclusions below) inside the guest and + emits it base64-encoded over `acq exec` stdout; the host decodes it into + `/paseo-state--.tar` (default dest `~/.acq-paseo-backups`). + It writes to a temp file and atomically renames on success, and verifies the + decoded result is a valid tar before promoting it — an interrupted capture + never leaves a truncated archive. +- **`paseo-restore [--apply] [backup-tar]`** prunes legacy runtime logs + from older backups, base64-encodes the backup tar on the host, pipes the text + into the guest which decodes and extracts it into a temporary sibling directory, + atomically swaps that directory into `${PASEO_HOME:-$HOME/.paseo}`, then bounces + the daemon so it re-reads the restored config/projects. With no `backup-tar` it + auto-selects the most recent backup for that sandbox. +- Both **default to dry-run**; `--apply` is required to write or mutate anything. + +## Transport is base64 text, not raw binary + +`acq` must work over **both** isolation backends (msb and sbx). Streaming a raw +binary tar over `acq exec` stdin/stdout was observed to **hang** on msb: a guest +`cat | wc -c` fed the tar bytes never saw EOF, so `tar -xf -` blocked forever. +Plain multi-line **text** stdin, by contrast, reaches EOF normally (verified). +`msb exec` exposes a `--stream` flag for byte-faithful binary I/O, but relying on +it would be backend-specific. + +So the archive is transported as **base64 text** in both directions — the guest +encodes on backup, the host encodes on restore — keeping the payload in the safe +text channel that behaves identically on every backend. `base64` is required on +both the host and in the guest image (present on the opencode base image). The +cost is ~33% transport inflation, which is immaterial once the snapshot is kept +small (see exclusions). This is why excluding the model cache matters: base64 of +a ~460MB model cache is ~640MB of text piped through `acq exec`, which crawled to +an apparent hang — trimming the snapshot to real state keeps the text channel +fast. + +## Capture is race-tolerant: snapshot-in-guest, then classify the tar exit code + +`acq exec` **faithfully propagates the guest command's exit code** (verified: +`exit 7` in the guest surfaces as host exit 7). That is a feature — it lets the +scripts detect real failures — but it means a naive `tar -cf - .` fails +on an *expected, non-fatal* condition: the daemon writes under `$PASEO_HOME` +while tar reads it, and GNU tar exits **1** ("file changed as we read it"), or +**2** when a whole entry is affected, *while still producing a valid archive of +everything else*. The first live `--apply` failed with exactly this (`rc=2`). + +The capture therefore does two things in the guest: + +1. **Snapshots to a guest temp file first**, then `cat`s that settled file to + stdout — so the bytes streamed to the host can't race the daemon. +2. **Classifies tar's exit code**: `0` = clean; `1`/`2` = files + changed/vanished mid-read (expected here — the archive is usable); `>= 3` = + a genuine failure (missing dir, write error), which aborts. `daemon.log` (the + busiest file) is excluded outright, and `--warning=no-file-changed + --ignore-failed-read` quiet the remaining churn. + +Because the guest capture now exits non-zero **only** on a real failure, the host +side *can* trust the propagated exit code, and additionally verifies the result +is a non-empty, valid tar (`tar -tf`) before atomically renaming it into place. +`paseo-restore` likewise checks both the propagated exit code and a completion +marker the guest prints, so a partial extract is never mistaken for success. + +## What the snapshot includes and excludes + +- **Included:** the `$PASEO_HOME` tree (`config.json`, `projects/`, `agents/`, + session records, anything else Paseo persists there) minus the specific + exclusions below. Capturing the tree broadly — rather than an allowlist of + known files — means a future Paseo state file is preserved automatically. +- **Excluded — `paseo.pid`:** a supervisor PID lock. Replaying a stale PID into a + fresh container is misleading; the daemon recreates it on boot. (Paseo's lock is + stale-tolerant, so a leftover would be reclaimed anyway, but there is no reason + to carry it.) +- **Excluded — `daemon.log`, `daemon.log.*`, `daemon.log.txt`, and + `*-daemon.log`:** live or rotated daemon logs can be multi-MB and are not + restorable state. Because live logs are appended *while the tar reads them*, GNU + tar reports "file changed as we read it" and exits non-zero (rc 1, or rc 2 when + a whole entry is affected) — the observed cause of the first live backup + failing. Rotated logs have also made restore appear hung because they can be + streamed before useful state files over acq's base64 text channel. +- **Excluded — `daemon-keypair.json`, `cli-client-id`, `server-id`:** + per-daemon/-install identity that the daemon regenerates on first boot. + Carrying it into a *different* sandbox is pointless at best and confusing at + worst. +- **Excluded — `models/`:** downloaded model caches (e.g. a ~460 MB sherpa-onnx + speech model observed on a live sandbox). Large, re-downloadable, and not + state; including it both bloats the archive and makes the base64 transport + crawl. The daemon re-fetches on demand. +- **Not captured — `~/.local/state/paseo`:** logs, install log, CA bundle. All + regenerated every boot. +- **Not captured — git worktree contents.** See below. + +## Why worktrees are left on the project mount, not moved into the backup + +The kit pins Paseo's `worktrees.root` to `/.paseo-worktrees`, +i.e. **inside a host bind mount** (see `worktrees-root-global-only.md`). A git +worktree stores absolute paths in both link halves (the worktree's `.git` file +points at `/.git/worktrees/`, which points back at the worktree's +absolute path). Because both the worktrees and their main repos live on the same +host mount, and `acq` re-mounts a project at the **same guest path** on recreate, +the worktree files and their git linkage **already survive recreation on their +own**. Moving worktrees into `$PASEO_HOME` so they would ride the backup tar would +break that linkage (the main repo would not move with them) and would make them +inaccessible from the host while active. + +So the real gap on recreate is not the worktree files — it is Paseo's *knowledge* +of them (its session/worktree records under `$PASEO_HOME`). Restoring the +`$PASEO_HOME` snapshot is exactly what makes Paseo re-recognize the intact on-disk +worktrees. This keeps the design simple and avoids leaving either the host or +Paseo in an inconsistent state. + +## The restore daemon bounce must clear the stale PID lock + +After extracting the snapshot, `paseo-restore` restarts the daemon so it re-reads +the restored `config.json` and projects. The kit runs the daemon under a +`supervisor:paseo-daemon` respawn loop, so the bounce **kills the daemon child** +(never the supervisor) and lets the supervisor relaunch it. + +But killing the daemon leaves its `$PASEO_HOME/paseo.pid` lock behind, and +Paseo only treats that lock as stale after **five minutes**. The supervisor +respawns every ~5 s, so without intervention every relaunch loses the race — +`Failed to acquire PID lock due to race condition` on repeat, and the daemon +never comes back (observed live). The bounce therefore **kills the daemon, waits +for it to actually exit, then removes `paseo.pid`** so the next respawn acquires +the lock cleanly. The lock is meaningless once its owner is dead, so removing it +is safe. (This is also why `paseo.pid` is excluded from the snapshot — a restored +stale lock would reintroduce the same stall.) + +## `worktrees.root` reconciliation on restore + +A restored `config.json` carries the **old** sandbox's absolute `worktrees.root`. +On the new sandbox the kit's `opencode` wrapper re-pins `worktrees.root` to +`/.paseo-worktrees` on the next `acq run` and bounces the daemon +(see `supervisor-owns-daemon-wrapper-pins-worktrees.md`), so a stale value +self-corrects. When restoring into a detached `acq create`, the user runs +`acq run ` once (documented in the script output, README, and +TROUBLESHOOTING) to trigger the re-pin. + +## The filename contract + +`paseo-backup` writes `paseo-state--.tar` where `` is a +colon-free ISO-8601 basic timestamp (`YYYYMMDDTHHMMSSZ`) so the name is valid on +every filesystem **and** sorts lexicographically by time. `paseo-restore`'s +"most recent backup" default relies on that ordering (lexical sort, last entry, +filtered to the sandbox prefix). The two scripts must change this format in +lockstep; both headers state the coupling. + +## Consequences + +- Recreation becomes a documented three-step ritual: `paseo-backup --apply` → + recreate → `paseo-restore --apply` (then one `acq run` to re-pin worktrees). +- No dependency on unshipped `acq` volume semantics; entirely inside this kit. +- Backups are plain tars on the host, inspectable with `tar -tf` and portable + between sandboxes if desired. +- Manual by design (per the current scope): there is no automatic + backup-on-teardown and no one-shot recreate wrapper. Those were considered and + deliberately not built now. +- Sandbox names flow into a filename glob and an `acq exec` target, so both + scripts constrain the name to `[A-Za-z0-9._-]` (SI-10). + +## Links + +- `worktrees-root-global-only.md` — why worktrees live on the project mount and + why `worktrees.root` is re-pinned per run. +- `supervisor-owns-daemon-wrapper-pins-worktrees.md` — the marker-scoped daemon + bounce that `paseo-restore` reuses. +- `../../../docs/decisions/0001-neutral-hybrid-v1-acq-kits.md` — the `volumes` + amendment whose creation-time/unseeded/per-sandbox semantics ruled it out here. +- `../../files/home/paseo-set-worktrees-root.mjs`, + `../../files/home/paseo-agent-shim` — the `${PASEO_HOME:-$HOME/.paseo}` + resolution the scripts mirror. +- `../../scripts/paseo-backup`, `../../scripts/paseo-restore`. diff --git a/integrations/isolation/acq-kits/paseo/scripts/paseo-backup b/integrations/isolation/acq-kits/paseo/scripts/paseo-backup new file mode 100755 index 00000000..276e912b --- /dev/null +++ b/integrations/isolation/acq-kits/paseo/scripts/paseo-backup @@ -0,0 +1,306 @@ +#!/usr/bin/env bash +# +# paseo-backup — snapshot a running sandbox's Paseo state to a host tar so it +# survives sandbox recreation (`acq rm` + a fresh `acq create`/`acq run`). +# +# WHAT THIS SOLVES: an acq sandbox is an ephemeral container. Everything under +# the guest home — including $PASEO_HOME (default ~/.paseo): config.json, the +# projects registry (projects/projects.json), and Paseo's session/agent metadata +# — is LOST when the sandbox is destroyed. Only host bind mounts survive. This +# script copies $PASEO_HOME out to the host BEFORE teardown; the companion +# `paseo-restore` replays it into a new sandbox afterward. +# +# WHAT IS AND IS NOT CAPTURED: +# - Captured: the $PASEO_HOME tree (config.json, projects/, agents/, session +# records, and any other state Paseo persists there) MINUS the exclusions +# below. +# - Excluded: +# * paseo.pid — supervisor PID lock; the daemon recreates it on boot. +# * daemon.log and rotated daemon logs — live/runtime noise that can bloat +# the archive and make restore look hung over acq's text transport. +# * daemon-keypair.json, cli-client-id, server-id — per-daemon/-install +# identity the daemon regenerates on first boot; carrying it into a +# DIFFERENT sandbox is pointless at best. +# * models/ — downloaded model caches (e.g. a ~460MB speech model). Large, +# re-downloadable, and not state; excluding it keeps the archive small. +# - NOT captured: git worktree CONTENTS. The kit pins Paseo's worktrees under +# /.paseo-worktrees, i.e. inside a host bind mount, so the +# worktree files ALREADY survive recreation on their own (as long as the +# project is re-mounted at the same guest path). What is lost without this +# backup is Paseo's KNOWLEDGE of them (its session/worktree records under +# $PASEO_HOME) — restoring this snapshot is exactly what makes Paseo +# re-recognize the on-disk worktrees. +# See docs/decisions/backup-restore-paseo-state.md. +# +# FILENAME CONTRACT (consumed by paseo-restore's newest-backup default): +# /paseo-state--.tar +# where is a colon-free ISO-8601 basic timestamp (YYYYMMDDTHHMMSSZ) that +# sorts lexicographically by time. paseo-restore picks the lexicographically +# last file matching this sandbox prefix as "most recent", so DO NOT change this +# naming without updating paseo-restore in lockstep. +# +# SAFETY: dry-run by default. It only reports what it WOULD capture until you +# pass --apply. Even the capture is read-only against the sandbox (a `tar -cf -` +# streamed over `acq exec` stdout); it never mutates guest state. +# +# Usage: +# paseo-backup [--apply] [dest-dir] +# paseo-backup --help +# +# Requires the `acq` CLI on PATH and a RUNNING sandbox named . + +set -euo pipefail +IFS=$'\n\t' + +APPLY=0 +SANDBOX="" +DEST_DIR="" +DEFAULT_DEST="${HOME}/.acq-paseo-backups" +TMP_OUT="" +TMP_ERR="" +TMP_B64="" + +usage() { + cat <<'EOF' +Usage: paseo-backup [--apply] [dest-dir] + +Snapshot a running acq sandbox's Paseo state ($PASEO_HOME, minus its PID lock, +live log, and per-install identity files) to a host tar so it survives sandbox +recreation. + +Arguments: + Name of the running acq sandbox to back up. + [dest-dir] Host directory to write the tar into. + Default: ~/.acq-paseo-backups + +Options: + --apply Actually create the backup. Without it, this is a dry run that + only prints what it would do. + --help Show this help. + +Output file (naming is a contract with paseo-restore): + /paseo-state--.tar (UTC = YYYYMMDDTHHMMSSZ) +EOF +} + +# Sandbox name flows into a filename and an `acq exec` target, so constrain it to +# a safe charset rather than trusting arbitrary input (SI-10). +valid_sandbox_name() { + case "$1" in + "" | *[!A-Za-z0-9._-]*) return 1 ;; + *) return 0 ;; + esac +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --apply) + APPLY=1 + shift + ;; + --help | -h) + usage + exit 0 + ;; + -*) + echo "Error: unknown option: $1" >&2 + usage >&2 + exit 2 + ;; + *) + if [[ -z "${SANDBOX}" ]]; then + SANDBOX="$1" + elif [[ -z "${DEST_DIR}" ]]; then + DEST_DIR="$1" + else + echo "Error: unexpected extra argument: $1" >&2 + usage >&2 + exit 2 + fi + shift + ;; + esac +done + +if [[ -z "${SANDBOX}" ]]; then + echo "Error: is required." >&2 + usage >&2 + exit 2 +fi +if ! valid_sandbox_name "${SANDBOX}"; then + echo "Error: sandbox name '${SANDBOX}' has characters outside [A-Za-z0-9._-]." >&2 + exit 2 +fi +if ! command -v acq >/dev/null 2>&1; then + echo "Error: the 'acq' CLI is not on PATH." >&2 + exit 2 +fi + +DEST_DIR="${DEST_DIR:-${DEFAULT_DEST}}" + +# Colon-free UTC timestamp so the filename is valid on every filesystem AND sorts +# lexicographically by time (paseo-restore relies on that ordering). +STAMP="$(date -u +%Y%m%dT%H%M%SZ)" +OUT_FILE="${DEST_DIR}/paseo-state-${SANDBOX}-${STAMP}.tar" + +# The guest-side capture. Resolve $PASEO_HOME exactly like the kit's own helpers +# (${PASEO_HOME:-$HOME/.paseo}) and emit a base64-encoded tar of its contents to +# stdout. Read-only against the sandbox. +# +# WHY THIS IS NOT A PLAIN `tar -cf - .`: two independent problems, both fixed +# here so the capture is backend-agnostic (works over msb AND sbx via acq). +# +# 1. LIVE-FILE CHURN. The daemon writes under $PASEO_HOME while we read it, and +# GNU tar treats a file changing mid-read as an error — exit 1 ("file +# changed as we read it") or 2 for a whole entry — WHILE STILL emitting a +# valid archive of everything else. We observed rc=2 on a live sandbox. +# Fix: snapshot to a guest TEMP FILE with churn tolerated +# (--warning=no-file-changed, --ignore-failed-read), and classify tar's exit +# code (0 clean; 1/2 expected churn; >=3 real failure → abort). +# +# 2. BINARY STDIN/STDOUT OVER acq. `acq exec` was observed to hang on a RAW +# BINARY stream (a guest `cat | wc -c` never saw EOF), while multi-line TEXT +# reaches EOF normally. So we do NOT stream raw tar bytes: the guest +# base64-ENCODES the archive to stdout, and the host base64-DECODES it. The +# restore side mirrors this (host encodes, guest decodes). base64 keeps the +# payload in acq's safe text channel on both backends. +# +# EXCLUSIONS (paths are relative to the -C dir, i.e. $PASEO_HOME root): +# - paseo.pid — supervisor PID lock; recreated on boot. +# - daemon.log, daemon.log.*, *-daemon.log, daemon.log.txt — live/runtime +# log artifacts; pure noise and the most churn-prone files. +# - daemon-keypair.json, cli-client-id, server-id — per-daemon/-install +# identity the daemon regenerates on first boot; carrying +# it into a DIFFERENT sandbox is pointless at best. +# - models/ — downloaded model caches (e.g. the ~460MB sherpa-onnx +# speech model). Large, re-downloadable, NOT state. Leaving +# it in bloats the archive to hundreds of MB and makes the +# base64 transport crawl; the daemon re-fetches on demand. +GUEST_CAPTURE=' + set -eu + PH="${PASEO_HOME:-$HOME/.paseo}" + if [ ! -d "$PH" ]; then + echo "paseo-backup: $PH does not exist in the sandbox" >&2 + exit 3 + fi + TMP="$(mktemp "${TMPDIR:-/tmp}/paseo-backup.XXXXXX.tar")" + trap "rm -f \"$TMP\"" EXIT INT TERM + rc=0 + tar -C "$PH" \ + --warning=no-file-changed \ + --ignore-failed-read \ + --exclude ./paseo.pid \ + --exclude ./daemon.log \ + --exclude ./daemon.log.* \ + --exclude ./daemon.log.txt \ + --exclude ./*-daemon.log \ + --exclude ./daemon-keypair.json \ + --exclude ./cli-client-id \ + --exclude ./server-id \ + --exclude ./models \ + -cf "$TMP" . || rc=$? + # rc 0 = clean; 1/2 = files changed/vanished during read (expected here, the + # archive is still usable); >=3 = fatal — abort. + if [ "$rc" -ge 3 ]; then + echo "paseo-backup: tar failed with exit $rc" >&2 + exit "$rc" + fi + base64 < "$TMP" +' + +cleanup() { + if [[ -n "${TMP_OUT}" && -f "${TMP_OUT}" ]]; then + rm -f "${TMP_OUT}" + fi + if [[ -n "${TMP_ERR}" && -f "${TMP_ERR}" ]]; then + rm -f "${TMP_ERR}" + fi + if [[ -n "${TMP_B64}" && -f "${TMP_B64}" ]]; then + rm -f "${TMP_B64}" + fi +} +trap cleanup EXIT INT TERM + +if [[ "${APPLY}" -ne 1 ]]; then + cat </dev/null 2>&1; then + echo "Error: 'base64' is required on the host to decode the archive." >&2 + exit 2 +fi + +# The guest emits the archive as base64 TEXT (raw binary over acq stdin/stdout +# hangs — see the GUEST_CAPTURE header). Capture that text to a temp file, then +# decode it into the final .tar and atomically rename on success — an interrupted +# or failed capture never leaves a truncated .tar that paseo-restore might later +# pick as "most recent". +TMP_B64="$(mktemp "${DEST_DIR}/.paseo-state-${SANDBOX}.b64.XXXXXX")" +TMP_OUT="$(mktemp "${DEST_DIR}/.paseo-state-${SANDBOX}.tar.XXXXXX")" +# Capture the guest's stderr separately so we can surface the ACTUAL cause (e.g. +# "$PASEO_HOME does not exist", an acq error, a permission problem) rather than a +# generic guess. +TMP_ERR="$(mktemp "${DEST_DIR}/.paseo-state-${SANDBOX}.err.XXXXXX")" + +# Run the capture. acq faithfully propagates the guest exit code (verified), and +# the guest capture above exits 0 for the expected file-churn conditions and +# non-zero only for a real failure — so we CAN branch on the exit code. Redirect +# stdin from /dev/null so acq never blocks on a TTY; keep guest stderr for +# diagnostics. +acq_rc=0 +acq exec "${SANDBOX}" -- sh -c "${GUEST_CAPTURE}" \ + "${TMP_B64}" 2>"${TMP_ERR}" || acq_rc=$? + +if [[ "${acq_rc}" -ne 0 ]]; then + echo "Error: backing up '${SANDBOX}' failed (exit ${acq_rc}). The sandbox / acq reported:" >&2 + if [[ -s "${TMP_ERR}" ]]; then + sed 's/^/ | /' "${TMP_ERR}" >&2 + else + echo " | (no error output — is 'acq' on PATH and the sandbox running?)" >&2 + fi + exit 1 +fi + +if [[ ! -s "${TMP_B64}" ]]; then + echo "Error: captured an empty archive from '${SANDBOX}'. Nothing was written." >&2 + if [[ -s "${TMP_ERR}" ]]; then + echo "The sandbox / acq reported:" >&2 + sed 's/^/ | /' "${TMP_ERR}" >&2 + else + echo " (\$PASEO_HOME may be absent, or the sandbox may not be running.)" >&2 + fi + exit 1 +fi + +# Decode the base64 text into the real tar. +if ! base64 -d <"${TMP_B64}" >"${TMP_OUT}" 2>/dev/null; then + echo "Error: could not base64-decode the captured stream; refusing to keep it." >&2 + exit 1 +fi + +# Sanity-check that we actually got a readable tar before promoting it. +if ! tar -tf "${TMP_OUT}" >/dev/null 2>&1; then + echo "Error: decoded data is not a valid tar archive; refusing to keep it." >&2 + if [[ -s "${TMP_ERR}" ]]; then + echo "The sandbox / acq reported:" >&2 + sed 's/^/ | /' "${TMP_ERR}" >&2 + fi + exit 1 +fi + +mv "${TMP_OUT}" "${OUT_FILE}" +TMP_OUT="" + +BYTES="$(wc -c <"${OUT_FILE}" | tr -d ' ')" +echo "Backed up Paseo state from '${SANDBOX}' to:" +echo " ${OUT_FILE} (${BYTES} bytes)" +echo "Restore it into a new sandbox with: paseo-restore --apply ${SANDBOX}" diff --git a/integrations/isolation/acq-kits/paseo/scripts/paseo-restore b/integrations/isolation/acq-kits/paseo/scripts/paseo-restore new file mode 100755 index 00000000..cfcfb3d5 --- /dev/null +++ b/integrations/isolation/acq-kits/paseo/scripts/paseo-restore @@ -0,0 +1,325 @@ +#!/usr/bin/env bash +# +# paseo-restore — replay a Paseo-state backup (from paseo-backup) into a running +# sandbox after recreation, then bounce the daemon so it picks the restored state +# up. +# +# WHAT THIS SOLVES: after `acq rm` + a fresh `acq create`/`acq run`, the new +# sandbox starts with empty Paseo state ($PASEO_HOME): no config.json, no +# registered projects, no session/agent history, and the on-disk git worktrees +# (which persist on the host mount) show up orphaned because Paseo no longer has +# their records. This streams a paseo-backup tar back into $PASEO_HOME and +# restarts the daemon so it re-reads the restored config and re-recognizes the +# projects and worktrees. +# +# BACKUP SELECTION: pass an explicit backup tar, or omit it to auto-select the +# MOST RECENT backup for this sandbox in the default backup dir +# (~/.acq-paseo-backups). "Most recent" is the lexicographically last file +# matching paseo-state--*.tar — which is the newest by time because +# paseo-backup timestamps are colon-free UTC (YYYYMMDDTHHMMSSZ) and sort by time. +# This coupling is a contract with paseo-backup's filename format; keep them in +# lockstep. +# +# WORKTREES.ROOT RECONCILIATION: a restored config.json carries the OLD +# sandbox's absolute worktrees.root. On the new sandbox the kit's `opencode` +# wrapper re-pins worktrees.root to /.paseo-worktrees on the +# next `acq run` and bounces the daemon, so a stale value self-corrects. If you +# restored into a detached `acq create`, run `acq run ` once (or +# re-pin manually) so the root matches this sandbox's mount. As long as the +# project is re-mounted at the SAME guest path, the on-disk worktrees line up +# with the restored records. See docs/decisions/backup-restore-paseo-state.md. +# +# SAFETY: dry-run by default. It resolves and prints the backup it WOULD restore, +# the target $PASEO_HOME, and that it would bounce the daemon — but writes nothing +# and does not touch the sandbox until you pass --apply. +# +# Usage: +# paseo-restore [--apply] [backup-tar] +# paseo-restore --help +# +# Requires the `acq` CLI on PATH and a RUNNING sandbox named . + +set -euo pipefail +IFS=$'\n\t' + +APPLY=0 +SANDBOX="" +BACKUP_TAR="" +DEFAULT_DEST="${HOME}/.acq-paseo-backups" +TMP_FILTER_DIR="" +TMP_FILTERED_TAR="" + +usage() { + cat <<'EOF' +Usage: paseo-restore [--apply] [backup-tar] + +Restore a paseo-backup snapshot into a running acq sandbox and bounce the daemon +so it re-reads the restored Paseo state. + +Arguments: + Name of the running acq sandbox to restore into. + [backup-tar] Path to a backup .tar produced by paseo-backup. If omitted, the + most recent backup for in ~/.acq-paseo-backups is used. + +Options: + --apply Actually restore. Without it, this is a dry run that only prints + the resolved backup, target, and planned daemon bounce. + --help Show this help. + +After restoring into a detached `acq create`, run `acq run ` +once so worktrees.root is re-pinned to this sandbox's project mount. +EOF +} + +# Sandbox name reaches an `acq exec` target and a filename glob, so constrain it +# to a safe charset (SI-10). +valid_sandbox_name() { + case "$1" in + "" | *[!A-Za-z0-9._-]*) return 1 ;; + *) return 0 ;; + esac +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --apply) + APPLY=1 + shift + ;; + --help | -h) + usage + exit 0 + ;; + -*) + echo "Error: unknown option: $1" >&2 + usage >&2 + exit 2 + ;; + *) + if [[ -z "${SANDBOX}" ]]; then + SANDBOX="$1" + elif [[ -z "${BACKUP_TAR}" ]]; then + BACKUP_TAR="$1" + else + echo "Error: unexpected extra argument: $1" >&2 + usage >&2 + exit 2 + fi + shift + ;; + esac +done + +if [[ -z "${SANDBOX}" ]]; then + echo "Error: is required." >&2 + usage >&2 + exit 2 +fi +if ! valid_sandbox_name "${SANDBOX}"; then + echo "Error: sandbox name '${SANDBOX}' has characters outside [A-Za-z0-9._-]." >&2 + exit 2 +fi +if ! command -v acq >/dev/null 2>&1; then + echo "Error: the 'acq' CLI is not on PATH." >&2 + exit 2 +fi + +# Resolve the backup to restore. Explicit path wins; otherwise pick the newest +# matching backup for this sandbox by lexical filename order (time order, per the +# paseo-backup naming contract). +if [[ -z "${BACKUP_TAR}" ]]; then + if [[ ! -d "${DEFAULT_DEST}" ]]; then + echo "Error: no backup given and default backup dir does not exist:" >&2 + echo " ${DEFAULT_DEST}" >&2 + echo " Create one with: paseo-backup --apply ${SANDBOX}" >&2 + exit 1 + fi + # Deterministic newest-by-name selection. Sort ascending, take the last line; + # the colon-free UTC stamp makes lexical order equal time order. Restrict the + # glob to this exact sandbox's prefix. + BACKUP_TAR="$( + find "${DEFAULT_DEST}" -maxdepth 1 -type f \ + -name "paseo-state-${SANDBOX}-*.tar" 2>/dev/null | LC_ALL=C sort | tail -n 1 + )" + if [[ -z "${BACKUP_TAR}" ]]; then + echo "Error: no backups found for sandbox '${SANDBOX}' in ${DEFAULT_DEST}" >&2 + echo " (expected files named paseo-state-${SANDBOX}-.tar)." >&2 + echo " Create one with: paseo-backup --apply ${SANDBOX}" >&2 + exit 1 + fi + echo "Auto-selected most recent backup for '${SANDBOX}':" + echo " ${BACKUP_TAR}" +fi + +if [[ ! -f "${BACKUP_TAR}" ]]; then + echo "Error: backup tar not found: ${BACKUP_TAR}" >&2 + exit 1 +fi +if ! tar -tf "${BACKUP_TAR}" >/dev/null 2>&1; then + echo "Error: '${BACKUP_TAR}' is not a readable tar archive." >&2 + exit 1 +fi + +cleanup() { + if [[ -n "${TMP_FILTER_DIR}" && -d "${TMP_FILTER_DIR}" ]]; then + rm -rf "${TMP_FILTER_DIR}" + fi + if [[ -n "${TMP_FILTERED_TAR}" && -f "${TMP_FILTERED_TAR}" ]]; then + rm -f "${TMP_FILTERED_TAR}" + fi +} +trap cleanup EXIT INT TERM + +# The guest-side restore. Decode and extract into a sibling temp directory, then +# swap it into $PASEO_HOME only after the extract is complete. We transport the +# tar as BASE64 TEXT, never raw binary: `acq exec` was observed to hang on a +# binary stdin stream (a guest `cat | wc -c` never saw EOF), while multi-line TEXT +# stdin reaches EOF normally. base64 keeps the payload in the safe text channel. +# Prints a unique DONE marker on a completed extract; acq faithfully propagates +# the guest exit code. +GUEST_RESTORE=' + set -eu + PH="${PASEO_HOME:-$HOME/.paseo}" + parent="$(dirname "$PH")" + base="$(basename "$PH")" + mkdir -p "$parent" + tmp="$(mktemp -d "$parent/.${base}.restore.XXXXXX")" + old="" + cleanup_restore() { + [ -n "$tmp" ] && [ -d "$tmp" ] && rm -rf "$tmp" + if [ -n "$old" ] && [ -e "$old" ] && [ ! -e "$PH" ]; then + mv "$old" "$PH" 2>/dev/null || true + fi + } + trap cleanup_restore EXIT INT TERM + base64 -d | tar -C "$tmp" -xf - + if [ -e "$PH" ]; then + old="$(mktemp -d "$parent/.${base}.old.XXXXXX")" + rmdir "$old" + mv "$PH" "$old" + fi + if ! mv "$tmp" "$PH"; then + if [ -n "$old" ] && [ -e "$old" ] && [ ! -e "$PH" ]; then + mv "$old" "$PH" 2>/dev/null || true + fi + exit 1 + fi + tmp="" + [ -n "$old" ] && rm -rf "$old" + trap - EXIT INT TERM + echo "PASEO_RESTORE_EXTRACT_OK $PH" +' + +# The daemon bounce: kill the Paseo-owned `paseo daemon start` process tree (NEVER +# the kit's supervisor respawn loop, whose argv carries the marker +# "supervisor:paseo-daemon"), then REMOVE the stale PID lock so the supervisor's +# next relaunch can acquire it cleanly. +# +# Why the lock removal is required: killing the daemon leaves its `paseo.pid` +# behind, and Paseo's lock is only treated as stale after >5 minutes. The +# supervisor respawns every 5s, so without clearing the lock the fresh daemon +# loses the race every time — "Failed to acquire PID lock due to race condition" +# on repeat, and the daemon never comes back (observed live). We hold the kill → +# confirm the process is gone → delete paseo.pid, so the relaunch starts clean. +# The lock is meaningless once its owner is dead, so removing it is safe. +GUEST_BOUNCE=' + set -eu + PH="${PASEO_HOME:-$HOME/.paseo}" + # Kill the daemon tree, but never the supervisor loop. + for pid in $(pgrep -u "$(id -u)" -f "paseo daemon start" 2>/dev/null); do + if tr "\0" " " < "/proc/$pid/cmdline" 2>/dev/null | grep -q "supervisor:paseo-daemon"; then + continue + fi + kill "$pid" 2>/dev/null || true + done + # Wait briefly for the daemon (non-supervisor) to actually exit before clearing + # the lock, so we do not race a still-shutting-down daemon. + i=0 + while [ "$i" -lt 20 ]; do + still="" + for pid in $(pgrep -u "$(id -u)" -f "paseo daemon start" 2>/dev/null); do + tr "\0" " " < "/proc/$pid/cmdline" 2>/dev/null | grep -q "supervisor:paseo-daemon" && continue + still=1 + done + [ -z "$still" ] && break + i=$((i + 1)) + sleep 0.25 + done + rm -f "$PH/paseo.pid" + echo "paseo-restore: killed daemon + cleared PID lock; the supervisor will respawn it" +' + +if [[ "${APPLY}" -ne 1 ]]; then + cat <&2 + tar -xf "${BACKUP_TAR}" -C "${TMP_FILTER_DIR}" + find "${TMP_FILTER_DIR}" -type f \ + \( -name 'daemon.log' -o -name 'daemon.log.*' -o -name 'daemon.log.txt' -o -name '*-daemon.log' \) \ + -delete + tar -C "${TMP_FILTER_DIR}" -cf "${TMP_FILTERED_TAR}" . + BACKUP_TAR="${TMP_FILTERED_TAR}" +fi + +# Stream the archive into the guest as BASE64 TEXT. Raw binary on acq's stdin was +# observed to hang (guest never sees EOF); base64 text reaches EOF normally. We +# pipe `base64 | acq exec ... 'base64 -d | tar -x'` and write acq's output +# to a temp file (no `$(...)` around acq, which also held its channel open). acq +# faithfully propagates the guest exit code; we ALSO require the guest completion +# marker, since a partial extract is worse than a clean failure. +if ! command -v base64 >/dev/null 2>&1; then + echo "Error: 'base64' is required on the host to transport the archive." >&2 + exit 2 +fi +RESTORE_OUT_FILE="$(mktemp "${TMPDIR:-/tmp}/paseo-restore.XXXXXX")" +RESTORE_BYTES="$(wc -c <"${BACKUP_TAR}" | tr -d ' ')" +echo "paseo-restore: streaming ${RESTORE_BYTES} bytes into '${SANDBOX}' (waiting for completion marker)" >&2 +restore_rc=0 +base64 <"${BACKUP_TAR}" | acq exec "${SANDBOX}" -- sh -c "${GUEST_RESTORE}" \ + >"${RESTORE_OUT_FILE}" 2>&1 || restore_rc=$? +RESTORE_OUT="$(cat "${RESTORE_OUT_FILE}")" +rm -f "${RESTORE_OUT_FILE}" +if [[ "${restore_rc}" -ne 0 ]] || ! printf '%s\n' "${RESTORE_OUT}" | grep -q '^PASEO_RESTORE_EXTRACT_OK '; then + echo "Error: restore into '${SANDBOX}' did not complete (exit ${restore_rc}). The sandbox / acq reported:" >&2 + if [[ -n "${RESTORE_OUT}" ]]; then + printf '%s\n' "${RESTORE_OUT}" | sed 's/^/ | /' >&2 + else + echo " | (no output — is 'acq' on PATH and the sandbox running?)" >&2 + fi + exit 1 +fi + +# Bounce the daemon so the restored config/projects take effect. Best-effort: +# the state is already in place, so a bounce hiccup is a warning, not a failure. +acq exec "${SANDBOX}" -- sh -c "${GUEST_BOUNCE}" /dev/null 2>&1 || { + echo "Warning: could not signal a daemon restart. The restored state is in place;" >&2 + echo " restart the daemon manually (or recreate the sandbox) to load it." >&2 +} + +cat < +once, so worktrees.root is re-pinned to this sandbox's project mount. As long as +the project is mounted at the same guest path, your worktrees line up with the +restored records. +EOF diff --git a/integrations/isolation/acq-kits/paseo/scripts/verify b/integrations/isolation/acq-kits/paseo/scripts/verify old mode 100644 new mode 100755 index e8591d99..226f5d65 --- a/integrations/isolation/acq-kits/paseo/scripts/verify +++ b/integrations/isolation/acq-kits/paseo/scripts/verify @@ -184,6 +184,96 @@ else bad "paseo-register-mounts.mjs failed node --check" fi +# --- Offline check of the backup/restore host scripts (no sandbox needed). ----- +# Assert they parse, are executable, and (with 'acq' stubbed out) behave in +# dry-run by default — printing "[dry-run]" and writing/mutating NOTHING — plus +# the newest-backup auto-selection and the sandbox-name charset guard. This +# exercises the host-side logic without a live sandbox; the live round-trip +# (backup -> clear -> restore) runs later under RUN_ACQ. +info "1d. backup/restore host scripts (offline)" +BACKUP_SH="$KIT_DIR/scripts/paseo-backup" +RESTORE_SH="$KIT_DIR/scripts/paseo-restore" +for _s in "$BACKUP_SH" "$RESTORE_SH"; do + if [ ! -f "$_s" ]; then + bad "$(basename "$_s") missing at $_s" + elif [ ! -x "$_s" ]; then + bad "$(basename "$_s") is not executable" + elif bash -n "$_s" >/dev/null 2>&1; then + ok "$(basename "$_s") parses (bash -n)" + else + bad "$(basename "$_s") failed bash -n" + fi +done +if [ -x "$BACKUP_SH" ] && [ -x "$RESTORE_SH" ]; then + # Stub 'acq' so the scripts pass their CLI-present guard without a real sandbox. + _stubdir="$WORK/stub-bin"; mkdir -p "$_stubdir" + printf '#!/bin/sh\nexit 0\n' > "$_stubdir/acq"; chmod +x "$_stubdir/acq" + _hostbackups="$WORK/host-backups" + # Dry-run backup must announce dry-run and write no file. + _bout="$(PATH="$_stubdir:$PATH" HOME="$WORK/fakehome" "$BACKUP_SH" verify-sbx "$_hostbackups" 2>&1 || true)" + if printf '%s' "$_bout" | grep -q '\[dry-run\]' && [ ! -d "$_hostbackups" ]; then + ok "paseo-backup defaults to dry-run and writes nothing" + else + bad "paseo-backup dry-run did not behave (out='$_bout', dir-exists=$([ -d "$_hostbackups" ] && echo yes || echo no))" + fi + # Sandbox-name charset guard (SI-10): a bad name must be rejected non-zero. + if PATH="$_stubdir:$PATH" "$BACKUP_SH" 'bad name!' >/dev/null 2>&1; then + bad "paseo-backup accepted an unsafe sandbox name" + else + ok "paseo-backup rejects an unsafe sandbox name" + fi + # Restore auto-selects the NEWEST backup for a sandbox by lexical (=time) order. + # The script reads ~/.acq-paseo-backups, so seed that under a fake HOME. The + # seeds must be VALID (if empty) tar archives — the restore validity-checks the + # resolved tar before printing its dry-run plan. + mkdir -p "$WORK/fakehome/.acq-paseo-backups" + tar -cf "$WORK/fakehome/.acq-paseo-backups/paseo-state-verify-sbx-20200101T000000Z.tar" -T /dev/null 2>/dev/null + tar -cf "$WORK/fakehome/.acq-paseo-backups/paseo-state-verify-sbx-20990101T000000Z.tar" -T /dev/null 2>/dev/null + _rout="$(PATH="$_stubdir:$PATH" HOME="$WORK/fakehome" "$RESTORE_SH" verify-sbx 2>&1 || true)" + if printf '%s' "$_rout" | grep -q '20990101T000000Z' && printf '%s' "$_rout" | grep -q '\[dry-run\]'; then + ok "paseo-restore auto-selects the newest backup and defaults to dry-run" + else + bad "paseo-restore newest-select/dry-run did not behave (out='$_rout')" + fi + + _legacy_src="$WORK/legacy-restore-src" + _legacy_out="$WORK/legacy-restore-out" + _legacy_tar="$WORK/legacy-restore.tar" + mkdir -p "$_legacy_src/projects" "$_legacy_src/agents/example" "$_legacy_out" + printf 'noise' > "$_legacy_src/20260101-0000-01-daemon.log" + printf 'noise' > "$_legacy_src/daemon.log.txt" + printf '{"ok":true}\n' > "$_legacy_src/config.json" + printf '[]\n' > "$_legacy_src/projects/projects.json" + printf '[]\n' > "$_legacy_src/projects/workspaces.json" + printf '{}\n' > "$_legacy_src/agents/example/agent.json" + tar -C "$_legacy_src" -cf "$_legacy_tar" . + cat > "$_stubdir/acq" <<'SH' +#!/bin/sh +if [ "$1" = "exec" ]; then + while [ "$#" -gt 0 ] && [ "$1" != "--" ]; do + shift + done + [ "$#" -gt 0 ] && shift + case "$*" in + *'base64 -d'*) exec "$@" ;; + *) exit 0 ;; + esac +else + exit 0 +fi +SH + chmod +x "$_stubdir/acq" + _restore_apply_out="$(PATH="$_stubdir:$PATH" HOME="$WORK/fakehome" PASEO_HOME="$_legacy_out/.paseo" "$RESTORE_SH" --apply verify-sbx "$_legacy_tar" 2>&1 || true)" + if [ -f "$_legacy_out/.paseo/config.json" ] \ + && [ ! -e "$_legacy_out/.paseo/20260101-0000-01-daemon.log" ] \ + && [ ! -e "$_legacy_out/.paseo/daemon.log.txt" ] \ + && printf '%s' "$_restore_apply_out" | grep -q 'streaming'; then + ok "paseo-restore filters legacy logs and reports progress" + else + bad "paseo-restore legacy filter/progress failed (out='$_restore_apply_out')" + fi +fi + # --- Select the live driver. ------------------------------------------------- DRIVER="" if [ "$RUN_ACQ" = "1" ]; then @@ -818,7 +908,75 @@ else fi fi +info "11. Backup/restore round-trip (live, via $DRIVER)" +# Prove the host-side backup/restore scripts survive a real teardown-shaped +# scenario against THIS sandbox: back up $PASEO_HOME to the host, drop a sentinel +# into the guest's $PASEO_HOME and remove it (simulating a fresh, stateless +# sandbox where the sentinel is gone), then restore and confirm the sentinel +# comes back — i.e. the snapshot round-trips through the host tar and the guest +# extract. paseo.pid must NOT appear in the archive. +if [ -z "$daemon_up" ]; then + skip "backup/restore round-trip (daemon not up, so no $PASEO_HOME to snapshot)" +elif [ ! -x "$KIT_DIR/scripts/paseo-backup" ] || [ ! -x "$KIT_DIR/scripts/paseo-restore" ]; then + skip "backup/restore round-trip (scripts missing/not executable)" +else + # Guest paths. Resolve $PASEO_HOME the same way the scripts do. + _guest_ph="$(in_sbx 'printf %s "${PASEO_HOME:-$HOME/.paseo}"')" + _sentinel="verify-roundtrip-$$-$(date -u +%s)" + # Seed a sentinel file, fake paseo.pid, and rotated log artifacts so we can + # assert round-trip and exclusions. Fail-open: if we can't seed, skip rather + # than falsely fail. + _seeded="$(in_sbx "mkdir -p \"$_guest_ph\" && printf '%s' '$_sentinel' > \"$_guest_ph/$_sentinel\" && printf 'stale' > \"$_guest_ph/paseo.pid\" && printf 'noise' > \"$_guest_ph/20260101-0000-01-daemon.log\" && printf 'noise' > \"$_guest_ph/daemon.log.txt\" && echo seeded")" + + if [ "$_seeded" != "seeded" ]; then + skip "backup/restore round-trip (could not seed guest \$PASEO_HOME at $_guest_ph)" + else + _bkdir="$WORK/roundtrip-backups" + if ! "$KIT_DIR/scripts/paseo-backup" --apply "$SBX_NAME" "$_bkdir" >/dev/null 2>&1; then + bad "paseo-backup --apply failed against the live sandbox" + else + _tar="$(ls -1 "$_bkdir"/paseo-state-"$SBX_NAME"-*.tar 2>/dev/null | LC_ALL=C sort | tail -n 1)" + if [ -z "$_tar" ] || [ ! -s "$_tar" ]; then + bad "paseo-backup produced no non-empty tar in $_bkdir" + elif tar -tf "$_tar" 2>/dev/null | grep -Eq "^\./(paseo\.pid|20260101-0000-01-daemon\.log|daemon\.log\.txt)$"; then + bad "backup archive wrongly INCLUDES runtime-only files" + else + ok "paseo-backup produced a valid tar excluding runtime-only files" + # Remove the sentinel from the guest to simulate lost state, then restore. + in_sbx "rm -f \"$_guest_ph/$_sentinel\"" >/dev/null + _gone="$(in_sbx "[ -e \"$_guest_ph/$_sentinel\" ] && echo present || echo gone")" + if [ "$_gone" != "gone" ]; then + bad "could not clear the sentinel before restore (round-trip inconclusive)" + elif ! "$KIT_DIR/scripts/paseo-restore" --apply "$SBX_NAME" "$_tar" >/dev/null 2>&1; then + bad "paseo-restore --apply failed against the live sandbox" + else + _back="$(in_sbx "cat \"$_guest_ph/$_sentinel\" 2>/dev/null")" + if [ "$_back" = "$_sentinel" ]; then + ok "restore round-tripped the sentinel back into \$PASEO_HOME" + else + bad "sentinel not restored (got '$_back', expected '$_sentinel')" + fi + # The restore bounces the daemon; confirm it heals back up (bounded). + _rt_deadline=$(( $(date +%s) + 45 )) + _healed="" + while :; do + [ "$(in_sbx "$health_probe")" = "yes" ] && { _healed=yes; break; } + [ "$(date +%s)" -ge "$_rt_deadline" ] && break + sleep "$POLL_INTERVAL" + done + if [ -n "$_healed" ]; then + ok "daemon healed after the restore-triggered bounce" + else + bad "daemon did not come back within 45s after the restore bounce" + fi + fi + fi + fi + # Tidy the guest sentinel + fake pid regardless of outcome (fail-open). + in_sbx "rm -f \"$_guest_ph/$_sentinel\" \"$_guest_ph/paseo.pid\" \"$_guest_ph/20260101-0000-01-daemon.log\" \"$_guest_ph/daemon.log.txt\"" >/dev/null 2>&1 || true + fi +fi + info "Summary" [ "$fail" -eq 0 ] && printf ' \033[32mAll checks passed.\033[0m\n' || printf ' \033[31mSome checks failed — see notes above.\033[0m\n' - [ "$fail" -eq 0 ]