From 502ff79b6bd8e90347c7d849a00091965c943f3f Mon Sep 17 00:00:00 2001 From: Bret Mogilefsky Date: Thu, 17 Sep 2026 06:00:41 +0000 Subject: [PATCH 1/4] feat(paseo): add backup/restore scripts to preserve state across recreation An acq sandbox is ephemeral: only host bind mounts survive teardown, so recreating one loses $PASEO_HOME (config.json, the projects registry, and session/agent records) and leaves the on-disk worktrees orphaned in Paseo. Add two host-side helpers that snapshot Paseo state before teardown and replay it into the new sandbox afterward, using only acq exec stream I/O (no new acq capability, no persistent-volume primitive, no coupling of state to a project repo, no extra mount arg): - scripts/paseo-backup: streams a tar of $PASEO_HOME (minus paseo.pid) over acq exec stdout to a host tar; dry-run by default, atomic temp+rename, validates a non-empty tar before promoting. - scripts/paseo-restore: streams a backup tar back into $PASEO_HOME and bounces the daemon (marker-scoped kill) so it re-reads restored config/projects; with no tar given it auto-selects the newest backup for the sandbox. Worktrees stay pinned under the project mount (they already survive recreate); the restored $PASEO_HOME is what makes Paseo re-recognize them, and worktrees.root self-corrects on the next acq run. Includes an ADR (rationale over persistent volumes, in/out of the snapshot, the worktree analysis, the filename contract), README + TROUBLESHOOTING sections, and verify assertions (offline dry-run/newest-select/name-guard checks + a live backup->clear->restore round-trip under RUN_ACQ). Co-authored-by: OpenCode Agent --- .../isolation/acq-kits/paseo/README.md | 58 ++++- .../acq-kits/paseo/TROUBLESHOOTING.md | 49 ++++ .../decisions/backup-restore-paseo-state.md | 128 ++++++++++ .../acq-kits/paseo/scripts/paseo-backup | 203 ++++++++++++++++ .../acq-kits/paseo/scripts/paseo-restore | 223 ++++++++++++++++++ .../isolation/acq-kits/paseo/scripts/verify | 121 +++++++++- 6 files changed, 779 insertions(+), 3 deletions(-) create mode 100644 integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md create mode 100755 integrations/isolation/acq-kits/paseo/scripts/paseo-backup create mode 100755 integrations/isolation/acq-kits/paseo/scripts/paseo-restore diff --git a/integrations/isolation/acq-kits/paseo/README.md b/integrations/isolation/acq-kits/paseo/README.md index 8c9a2cb7..d5f4a9bb 100644 --- a/integrations/isolation/acq-kits/paseo/README.md +++ b/integrations/isolation/acq-kits/paseo/README.md @@ -141,6 +141,56 @@ To override the location, pre-set `worktrees.root` to an absolute path in replaced on the next run — set it via the entrypoint's working directory instead, or remove the shim's pinning if you need a custom fixed root. +## Backing up and restoring Paseo state across recreation + +An `acq` sandbox is ephemeral: only your host bind mounts survive its +destruction. The whole guest home — including `$PASEO_HOME` (default `~/.paseo`: +`config.json`, the projects registry, and Paseo's session/agent records) — is +**lost** when you `acq rm` and recreate. Your git worktrees themselves survive +(they live under `/.paseo-worktrees` on the host mount), but after a +recreate Paseo no longer has their records, so they show up orphaned. + +Two host-side helper scripts snapshot Paseo state before teardown and replay it +into the new sandbox afterward: + +```bash +# Before recreating — snapshot state to the host (default ~/.acq-paseo-backups). +# Dry-run first (prints what it would do), then --apply to actually write it: +scripts/paseo-backup +scripts/paseo-backup --apply + +# Recreate the sandbox as usual: +acq rm +acq run opencode /path/to/your/project # (or acq create …) + +# After the new sandbox is up — restore. With no tar given it auto-selects the +# most recent backup for that sandbox. Dry-run first, then --apply: +scripts/paseo-restore +scripts/paseo-restore --apply +``` + +`paseo-restore` streams the snapshot back into `$PASEO_HOME` and restarts the +daemon so it re-reads the restored config and re-recognizes your projects and +worktrees. + +- **What's captured:** the full `$PASEO_HOME` tree **minus `paseo.pid`** (a + supervisor PID lock the daemon recreates on boot). Logs under + `~/.local/state/paseo` are regenerated every boot and are not captured. +- **Worktrees:** their contents are **not** in the backup — they already persist + on the host mount. The restored `$PASEO_HOME` is what makes Paseo re-recognize + them, as long as the project is re-mounted at the **same guest path** (acq's + normal behavior). +- **`worktrees.root`:** a restored `config.json` carries the old sandbox's root. + It self-corrects on the next `acq run` (the wrapper re-pins it). If you + restored into a **detached** `acq create`, run `acq run opencode ` + once so the root matches this sandbox's mount. +- **Safety:** both scripts default to dry-run; pass `--apply` to actually write + or restore. The backup is read-only against the sandbox. + +Backups are plain tars — inspect one with `tar -tf `. See +`docs/decisions/backup-restore-paseo-state.md` for the design and the worktree +rationale. + ## No shared session with a terminal TUI Unlike a shared-server setup (e.g. the `openchamber` kit), Paseo is **not** an @@ -239,6 +289,8 @@ paseo/ ├── README.md # this file ├── TROUBLESHOOTING.md # failure modes and fixes ├── scripts/verify # host-side live check +├── scripts/paseo-backup # snapshot $PASEO_HOME to a host tar (dry-run default) +├── scripts/paseo-restore # replay a snapshot into a new sandbox + bounce the daemon └── docs/decisions/ # design records (ADRs) ``` @@ -248,5 +300,7 @@ Rationale and the decisions behind this kit's structure live in [`docs/decisions/`](docs/decisions/) — notably why one daemon serves everything on a single port, why worktrees can only be pinned to the first project dir (and why that needs a daemon restart), why the startup script owns the daemon while -the entrypoint pins worktrees, why the install runs at startup, and why the -entrypoint is split into a thin agent-named wrapper over a generic kit shim. +the entrypoint pins worktrees, why the install runs at startup, why the +entrypoint is split into a thin agent-named wrapper over a generic kit shim, and +why Paseo state is preserved across sandbox recreation with backup/restore +scripts rather than a persistent volume. diff --git a/integrations/isolation/acq-kits/paseo/TROUBLESHOOTING.md b/integrations/isolation/acq-kits/paseo/TROUBLESHOOTING.md index d5a62788..a4b4463f 100644 --- a/integrations/isolation/acq-kits/paseo/TROUBLESHOOTING.md +++ b/integrations/isolation/acq-kits/paseo/TROUBLESHOOTING.md @@ -328,6 +328,55 @@ rewrites `config.json` when the value differs. If you need a custom fixed root, don't rely on the wrapper's pin — the current kit does not expose an override env var. +## I recreated the sandbox and lost my Paseo state / my worktrees show as orphaned + +**Symptom.** After `acq rm` + a fresh `acq create`/`acq run`, the Paseo UI has no +projects, no session history, and any git worktrees under +`/.paseo-worktrees` are present on disk but unknown to Paseo. + +**Cause.** The sandbox is ephemeral — only host bind mounts survive. `$PASEO_HOME` +(default `~/.paseo`: `config.json`, the projects registry, session/agent records) +is lost on recreate. The worktree *files* persist (they live on the host mount), +but Paseo's *records* of them do not, so they look orphaned. + +**Prevention + recovery.** Use the kit's host-side scripts. Snapshot before you +recreate, restore after the new sandbox is up: + +```bash +# BEFORE recreating (dry-run prints what it would do; --apply writes it): +scripts/paseo-backup --apply # → ~/.acq-paseo-backups/paseo-state--.tar + +# recreate as usual … +acq rm +acq run opencode /path/to/your/project + +# AFTER the new sandbox is up (auto-selects the most recent backup for it): +scripts/paseo-restore --apply +``` + +`paseo-restore` streams the snapshot back into `$PASEO_HOME` and bounces the +daemon so it re-reads the restored config and re-recognizes your projects and +worktrees. Confirm it took: + +```bash +acq exec -- sh -c 'cat "${PASEO_HOME:-$HOME/.paseo}/config.json"; ls "${PASEO_HOME:-$HOME/.paseo}/projects" 2>/dev/null' +acq exec -- sh -c 'curl -fsS http://127.0.0.1:6767/api/health && echo OK' +``` + +Notes: + +- **You must have made a backup first** — restore replays a prior snapshot; it + cannot recover state that was never captured. Recreation is unplanned by + definition, so back up proactively when your Paseo state matters. +- **`worktrees.root` self-corrects on the next `acq run`** (the wrapper re-pins + it). If you restored into a **detached** `acq create`, run + `acq run opencode ` once so the root matches this sandbox's mount. +- **Same guest path required.** Worktrees line up with the restored records only + if the project is re-mounted at the same guest path (acq's normal behavior). +- **`paseo.pid` is intentionally not in the backup**; the daemon recreates it. + +See `docs/decisions/backup-restore-paseo-state.md` for the design. + ## Endless `relay_error` / `relay_control_disconnected` in the daemon log **Symptom.** The daemon log fills with, every ~30s: diff --git a/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md b/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md new file mode 100644 index 00000000..6699e954 --- /dev/null +++ b/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md @@ -0,0 +1,128 @@ +# Decision: back up and restore Paseo state across sandbox recreation + +**Status:** accepted + +## Context + +An `acq` sandbox is an ephemeral container. Only the host bind mounts survive its +destruction; the entire guest filesystem — including `$PASEO_HOME` (default +`~/.paseo`: `config.json`, the `projects/projects.json` registry, and Paseo's +session/agent records) and `~/.local/state/paseo` (logs) — is lost on +`acq rm` + a fresh create. So every unplanned sandbox recreation drops the user's +Paseo state, and the on-disk git worktrees that DO persist (see below) show up +"orphaned" in the UI because Paseo no longer holds their records. + +We evaluated three durable-storage approaches before this one: + +1. **Relocate `$PASEO_HOME` onto the first project mount.** Survives recreate + today, but couples Paseo's home to a repo directory (needs `.gitignore` + hygiene) and scopes state to one project. +2. **A dedicated host state mount** (`PASEO_HOME` on an extra mounted dir). + Clean, but forces an extra mount argument on every `acq run`/`acq create`. +3. **A kit-declared persistent volume.** The `hybrid/v1` schema already has a + `volumes` block, but its documented semantics are the opposite of what is + needed here: it is **creation-time only, mounts UNSEEDED, and is per-sandbox + scratch storage** — it does not outlive the sandbox, and the translate layer + that maps it lives in the separate `acq`/quickstart repo, not this one. It + therefore cannot provide cross-recreate persistence without an upstream + `acq`-side change (see the `volumes` amendment in + `../../../docs/decisions/0001-neutral-hybrid-v1-acq-kits.md`). + +## Decision + +**Ship two host-side helper scripts — `scripts/paseo-backup` and +`scripts/paseo-restore` — that snapshot `$PASEO_HOME` to a host tar before +teardown and replay it into a new sandbox afterward.** They use only `acq exec` +stdin/stdout streaming (already the kit's universal in-guest mechanism); no new +`acq` capability, no persistent-storage primitive, no coupling of Paseo state to +a project repo, and no extra mount argument. + +- **`paseo-backup [--apply] [dest-dir]`** streams + `tar -C "${PASEO_HOME:-$HOME/.paseo}" --exclude ./paseo.pid -cf - .` over + `acq exec` stdout into `/paseo-state--.tar` + (default dest `~/.acq-paseo-backups`). It writes to a temp file and atomically + renames on success, and verifies the captured stream is a non-empty, valid tar + before promoting it — an interrupted capture never leaves a truncated archive. +- **`paseo-restore [--apply] [backup-tar]`** streams a backup tar into + `${PASEO_HOME:-$HOME/.paseo}` in the target sandbox, then bounces the daemon so + it re-reads the restored config/projects. With no `backup-tar` it auto-selects + the most recent backup for that sandbox. +- Both **default to dry-run**; `--apply` is required to write or mutate anything. + +## What the snapshot includes and excludes + +- **Included:** the full `$PASEO_HOME` tree (`config.json`, `projects/`, + session/agent records, anything else Paseo persists there). Capturing the whole + tree — rather than an allowlist of known files — means a future Paseo state file + is preserved automatically. +- **Excluded — `paseo.pid`:** a supervisor PID lock. Replaying a stale PID into a + fresh container is misleading; the daemon recreates it on boot. (Paseo's lock is + stale-tolerant, so a leftover would be reclaimed anyway, but there is no reason + to carry it.) +- **Not captured — `~/.local/state/paseo`:** logs, install log, CA bundle. All + regenerated every boot. +- **Not captured — git worktree contents.** See below. + +## Why worktrees are left on the project mount, not moved into the backup + +The kit pins Paseo's `worktrees.root` to `/.paseo-worktrees`, +i.e. **inside a host bind mount** (see `worktrees-root-global-only.md`). A git +worktree stores absolute paths in both link halves (the worktree's `.git` file +points at `/.git/worktrees/`, which points back at the worktree's +absolute path). Because both the worktrees and their main repos live on the same +host mount, and `acq` re-mounts a project at the **same guest path** on recreate, +the worktree files and their git linkage **already survive recreation on their +own**. Moving worktrees into `$PASEO_HOME` so they would ride the backup tar would +break that linkage (the main repo would not move with them) and would make them +inaccessible from the host while active. + +So the real gap on recreate is not the worktree files — it is Paseo's *knowledge* +of them (its session/worktree records under `$PASEO_HOME`). Restoring the +`$PASEO_HOME` snapshot is exactly what makes Paseo re-recognize the intact on-disk +worktrees. This keeps the design simple and avoids leaving either the host or +Paseo in an inconsistent state. + +## `worktrees.root` reconciliation on restore + +A restored `config.json` carries the **old** sandbox's absolute `worktrees.root`. +On the new sandbox the kit's `opencode` wrapper re-pins `worktrees.root` to +`/.paseo-worktrees` on the next `acq run` and bounces the daemon +(see `supervisor-owns-daemon-wrapper-pins-worktrees.md`), so a stale value +self-corrects. When restoring into a detached `acq create`, the user runs +`acq run ` once (documented in the script output, README, and +TROUBLESHOOTING) to trigger the re-pin. + +## The filename contract + +`paseo-backup` writes `paseo-state--.tar` where `` is a +colon-free ISO-8601 basic timestamp (`YYYYMMDDTHHMMSSZ`) so the name is valid on +every filesystem **and** sorts lexicographically by time. `paseo-restore`'s +"most recent backup" default relies on that ordering (lexical sort, last entry, +filtered to the sandbox prefix). The two scripts must change this format in +lockstep; both headers state the coupling. + +## Consequences + +- Recreation becomes a documented three-step ritual: `paseo-backup --apply` → + recreate → `paseo-restore --apply` (then one `acq run` to re-pin worktrees). +- No dependency on unshipped `acq` volume semantics; entirely inside this kit. +- Backups are plain tars on the host, inspectable with `tar -tf` and portable + between sandboxes if desired. +- Manual by design (per the current scope): there is no automatic + backup-on-teardown and no one-shot recreate wrapper. Those were considered and + deliberately not built now. +- Sandbox names flow into a filename glob and an `acq exec` target, so both + scripts constrain the name to `[A-Za-z0-9._-]` (SI-10). + +## Links + +- `worktrees-root-global-only.md` — why worktrees live on the project mount and + why `worktrees.root` is re-pinned per run. +- `supervisor-owns-daemon-wrapper-pins-worktrees.md` — the marker-scoped daemon + bounce that `paseo-restore` reuses. +- `../../../docs/decisions/0001-neutral-hybrid-v1-acq-kits.md` — the `volumes` + amendment whose creation-time/unseeded/per-sandbox semantics ruled it out here. +- `../../files/home/paseo-set-worktrees-root.mjs`, + `../../files/home/paseo-agent-shim` — the `${PASEO_HOME:-$HOME/.paseo}` + resolution the scripts mirror. +- `../../scripts/paseo-backup`, `../../scripts/paseo-restore`. diff --git a/integrations/isolation/acq-kits/paseo/scripts/paseo-backup b/integrations/isolation/acq-kits/paseo/scripts/paseo-backup new file mode 100755 index 00000000..a9dc267d --- /dev/null +++ b/integrations/isolation/acq-kits/paseo/scripts/paseo-backup @@ -0,0 +1,203 @@ +#!/usr/bin/env bash +# +# paseo-backup — snapshot a running sandbox's Paseo state to a host tar so it +# survives sandbox recreation (`acq rm` + a fresh `acq create`/`acq run`). +# +# WHAT THIS SOLVES: an acq sandbox is an ephemeral container. Everything under +# the guest home — including $PASEO_HOME (default ~/.paseo): config.json, the +# projects registry (projects/projects.json), and Paseo's session/agent metadata +# — is LOST when the sandbox is destroyed. Only host bind mounts survive. This +# script copies $PASEO_HOME out to the host BEFORE teardown; the companion +# `paseo-restore` replays it into a new sandbox afterward. +# +# WHAT IS AND IS NOT CAPTURED: +# - Captured: the full $PASEO_HOME tree (config.json, projects/, session/agent +# records, any other state Paseo persists there) MINUS paseo.pid. +# - Excluded: paseo.pid — a supervisor PID lock. Replaying a stale PID into a +# fresh container would be misleading; the daemon recreates it on boot. +# - NOT captured: git worktree CONTENTS. The kit pins Paseo's worktrees under +# /.paseo-worktrees, i.e. inside a host bind mount, so the +# worktree files ALREADY survive recreation on their own (as long as the +# project is re-mounted at the same guest path). What is lost without this +# backup is Paseo's KNOWLEDGE of them (its session/worktree records under +# $PASEO_HOME) — restoring this snapshot is exactly what makes Paseo +# re-recognize the on-disk worktrees. Logs under ~/.local/state/paseo are +# regenerated every boot and are also not captured. +# See docs/decisions/backup-restore-paseo-state.md. +# +# FILENAME CONTRACT (consumed by paseo-restore's newest-backup default): +# /paseo-state--.tar +# where is a colon-free ISO-8601 basic timestamp (YYYYMMDDTHHMMSSZ) that +# sorts lexicographically by time. paseo-restore picks the lexicographically +# last file matching this sandbox prefix as "most recent", so DO NOT change this +# naming without updating paseo-restore in lockstep. +# +# SAFETY: dry-run by default. It only reports what it WOULD capture until you +# pass --apply. Even the capture is read-only against the sandbox (a `tar -cf -` +# streamed over `acq exec` stdout); it never mutates guest state. +# +# Usage: +# paseo-backup [--apply] [dest-dir] +# paseo-backup --help +# +# Requires the `acq` CLI on PATH and a RUNNING sandbox named . + +set -euo pipefail +IFS=$'\n\t' + +APPLY=0 +SANDBOX="" +DEST_DIR="" +DEFAULT_DEST="${HOME}/.acq-paseo-backups" +TMP_OUT="" + +usage() { + cat <<'EOF' +Usage: paseo-backup [--apply] [dest-dir] + +Snapshot a running acq sandbox's Paseo state ($PASEO_HOME, minus paseo.pid) to a +host tar so it survives sandbox recreation. + +Arguments: + Name of the running acq sandbox to back up. + [dest-dir] Host directory to write the tar into. + Default: ~/.acq-paseo-backups + +Options: + --apply Actually create the backup. Without it, this is a dry run that + only prints what it would do. + --help Show this help. + +Output file (naming is a contract with paseo-restore): + /paseo-state--.tar (UTC = YYYYMMDDTHHMMSSZ) +EOF +} + +# Sandbox name flows into a filename and an `acq exec` target, so constrain it to +# a safe charset rather than trusting arbitrary input (SI-10). +valid_sandbox_name() { + case "$1" in + "" | *[!A-Za-z0-9._-]*) return 1 ;; + *) return 0 ;; + esac +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --apply) + APPLY=1 + shift + ;; + --help | -h) + usage + exit 0 + ;; + -*) + echo "Error: unknown option: $1" >&2 + usage >&2 + exit 2 + ;; + *) + if [[ -z "${SANDBOX}" ]]; then + SANDBOX="$1" + elif [[ -z "${DEST_DIR}" ]]; then + DEST_DIR="$1" + else + echo "Error: unexpected extra argument: $1" >&2 + usage >&2 + exit 2 + fi + shift + ;; + esac +done + +if [[ -z "${SANDBOX}" ]]; then + echo "Error: is required." >&2 + usage >&2 + exit 2 +fi +if ! valid_sandbox_name "${SANDBOX}"; then + echo "Error: sandbox name '${SANDBOX}' has characters outside [A-Za-z0-9._-]." >&2 + exit 2 +fi +if ! command -v acq >/dev/null 2>&1; then + echo "Error: the 'acq' CLI is not on PATH." >&2 + exit 2 +fi + +DEST_DIR="${DEST_DIR:-${DEFAULT_DEST}}" + +# Colon-free UTC timestamp so the filename is valid on every filesystem AND sorts +# lexicographically by time (paseo-restore relies on that ordering). +STAMP="$(date -u +%Y%m%dT%H%M%SZ)" +OUT_FILE="${DEST_DIR}/paseo-state-${SANDBOX}-${STAMP}.tar" + +# The guest-side capture. Resolve $PASEO_HOME exactly like the kit's own helpers +# (${PASEO_HOME:-$HOME/.paseo}) and stream a tar of its contents (minus the PID +# lock) to stdout. Read-only against the sandbox. `--exclude ./paseo.pid` is +# relative to the -C dir. If $PASEO_HOME does not exist yet, tar has nothing to +# archive and we detect the empty result below rather than write a bogus file. +GUEST_CAPTURE=' + set -eu + PH="${PASEO_HOME:-$HOME/.paseo}" + if [ ! -d "$PH" ]; then + echo "paseo-backup: $PH does not exist in the sandbox" >&2 + exit 3 + fi + tar -C "$PH" --exclude ./paseo.pid -cf - . +' + +cleanup() { + if [[ -n "${TMP_OUT}" && -f "${TMP_OUT}" ]]; then + rm -f "${TMP_OUT}" + fi +} +trap cleanup EXIT INT TERM + +if [[ "${APPLY}" -ne 1 ]]; then + cat <"${TMP_OUT}"; then + echo "Error: 'acq exec ${SANDBOX}' failed — is the sandbox running?" >&2 + exit 1 +fi + +if [[ ! -s "${TMP_OUT}" ]]; then + echo "Error: captured an empty archive — the sandbox may not be running or" >&2 + echo " \$PASEO_HOME is absent. Nothing was written." >&2 + exit 1 +fi + +# Sanity-check that we actually got a readable tar before promoting it. +if ! tar -tf "${TMP_OUT}" >/dev/null 2>&1; then + echo "Error: captured data is not a valid tar archive; refusing to keep it." >&2 + exit 1 +fi + +mv "${TMP_OUT}" "${OUT_FILE}" +TMP_OUT="" + +BYTES="$(wc -c <"${OUT_FILE}" | tr -d ' ')" +echo "Backed up Paseo state from '${SANDBOX}' to:" +echo " ${OUT_FILE} (${BYTES} bytes)" +echo "Restore it into a new sandbox with: paseo-restore --apply ${SANDBOX}" diff --git a/integrations/isolation/acq-kits/paseo/scripts/paseo-restore b/integrations/isolation/acq-kits/paseo/scripts/paseo-restore new file mode 100755 index 00000000..31c91e3d --- /dev/null +++ b/integrations/isolation/acq-kits/paseo/scripts/paseo-restore @@ -0,0 +1,223 @@ +#!/usr/bin/env bash +# +# paseo-restore — replay a Paseo-state backup (from paseo-backup) into a running +# sandbox after recreation, then bounce the daemon so it picks the restored state +# up. +# +# WHAT THIS SOLVES: after `acq rm` + a fresh `acq create`/`acq run`, the new +# sandbox starts with empty Paseo state ($PASEO_HOME): no config.json, no +# registered projects, no session/agent history, and the on-disk git worktrees +# (which persist on the host mount) show up orphaned because Paseo no longer has +# their records. This streams a paseo-backup tar back into $PASEO_HOME and +# restarts the daemon so it re-reads the restored config and re-recognizes the +# projects and worktrees. +# +# BACKUP SELECTION: pass an explicit backup tar, or omit it to auto-select the +# MOST RECENT backup for this sandbox in the default backup dir +# (~/.acq-paseo-backups). "Most recent" is the lexicographically last file +# matching paseo-state--*.tar — which is the newest by time because +# paseo-backup timestamps are colon-free UTC (YYYYMMDDTHHMMSSZ) and sort by time. +# This coupling is a contract with paseo-backup's filename format; keep them in +# lockstep. +# +# WORKTREES.ROOT RECONCILIATION: a restored config.json carries the OLD +# sandbox's absolute worktrees.root. On the new sandbox the kit's `opencode` +# wrapper re-pins worktrees.root to /.paseo-worktrees on the +# next `acq run` and bounces the daemon, so a stale value self-corrects. If you +# restored into a detached `acq create`, run `acq run ` once (or +# re-pin manually) so the root matches this sandbox's mount. As long as the +# project is re-mounted at the SAME guest path, the on-disk worktrees line up +# with the restored records. See docs/decisions/backup-restore-paseo-state.md. +# +# SAFETY: dry-run by default. It resolves and prints the backup it WOULD restore, +# the target $PASEO_HOME, and that it would bounce the daemon — but writes nothing +# and does not touch the sandbox until you pass --apply. +# +# Usage: +# paseo-restore [--apply] [backup-tar] +# paseo-restore --help +# +# Requires the `acq` CLI on PATH and a RUNNING sandbox named . + +set -euo pipefail +IFS=$'\n\t' + +APPLY=0 +SANDBOX="" +BACKUP_TAR="" +DEFAULT_DEST="${HOME}/.acq-paseo-backups" + +usage() { + cat <<'EOF' +Usage: paseo-restore [--apply] [backup-tar] + +Restore a paseo-backup snapshot into a running acq sandbox and bounce the daemon +so it re-reads the restored Paseo state. + +Arguments: + Name of the running acq sandbox to restore into. + [backup-tar] Path to a backup .tar produced by paseo-backup. If omitted, the + most recent backup for in ~/.acq-paseo-backups is used. + +Options: + --apply Actually restore. Without it, this is a dry run that only prints + the resolved backup, target, and planned daemon bounce. + --help Show this help. + +After restoring into a detached `acq create`, run `acq run ` +once so worktrees.root is re-pinned to this sandbox's project mount. +EOF +} + +# Sandbox name reaches an `acq exec` target and a filename glob, so constrain it +# to a safe charset (SI-10). +valid_sandbox_name() { + case "$1" in + "" | *[!A-Za-z0-9._-]*) return 1 ;; + *) return 0 ;; + esac +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --apply) + APPLY=1 + shift + ;; + --help | -h) + usage + exit 0 + ;; + -*) + echo "Error: unknown option: $1" >&2 + usage >&2 + exit 2 + ;; + *) + if [[ -z "${SANDBOX}" ]]; then + SANDBOX="$1" + elif [[ -z "${BACKUP_TAR}" ]]; then + BACKUP_TAR="$1" + else + echo "Error: unexpected extra argument: $1" >&2 + usage >&2 + exit 2 + fi + shift + ;; + esac +done + +if [[ -z "${SANDBOX}" ]]; then + echo "Error: is required." >&2 + usage >&2 + exit 2 +fi +if ! valid_sandbox_name "${SANDBOX}"; then + echo "Error: sandbox name '${SANDBOX}' has characters outside [A-Za-z0-9._-]." >&2 + exit 2 +fi +if ! command -v acq >/dev/null 2>&1; then + echo "Error: the 'acq' CLI is not on PATH." >&2 + exit 2 +fi + +# Resolve the backup to restore. Explicit path wins; otherwise pick the newest +# matching backup for this sandbox by lexical filename order (time order, per the +# paseo-backup naming contract). +if [[ -z "${BACKUP_TAR}" ]]; then + if [[ ! -d "${DEFAULT_DEST}" ]]; then + echo "Error: no backup given and default backup dir does not exist:" >&2 + echo " ${DEFAULT_DEST}" >&2 + echo " Create one with: paseo-backup --apply ${SANDBOX}" >&2 + exit 1 + fi + # Deterministic newest-by-name selection. Sort ascending, take the last line; + # the colon-free UTC stamp makes lexical order equal time order. Restrict the + # glob to this exact sandbox's prefix. + BACKUP_TAR="$( + find "${DEFAULT_DEST}" -maxdepth 1 -type f \ + -name "paseo-state-${SANDBOX}-*.tar" 2>/dev/null | LC_ALL=C sort | tail -n 1 + )" + if [[ -z "${BACKUP_TAR}" ]]; then + echo "Error: no backups found for sandbox '${SANDBOX}' in ${DEFAULT_DEST}" >&2 + echo " (expected files named paseo-state-${SANDBOX}-.tar)." >&2 + echo " Create one with: paseo-backup --apply ${SANDBOX}" >&2 + exit 1 + fi + echo "Auto-selected most recent backup for '${SANDBOX}':" + echo " ${BACKUP_TAR}" +fi + +if [[ ! -f "${BACKUP_TAR}" ]]; then + echo "Error: backup tar not found: ${BACKUP_TAR}" >&2 + exit 1 +fi +if ! tar -tf "${BACKUP_TAR}" >/dev/null 2>&1; then + echo "Error: '${BACKUP_TAR}' is not a readable tar archive." >&2 + exit 1 +fi + +# The guest-side restore. Recreate $PASEO_HOME if needed and extract the archive +# into it, then the caller bounces the daemon separately (below). Resolves +# $PASEO_HOME the same way the kit's helpers do. +GUEST_RESTORE=' + set -eu + PH="${PASEO_HOME:-$HOME/.paseo}" + mkdir -p "$PH" + tar -C "$PH" -xf - + echo "paseo-restore: extracted into $PH" +' + +# The daemon bounce: kill only the Paseo-owned `paseo daemon start` process tree, +# NEVER the kit's supervisor respawn loop (whose argv carries the marker +# "supervisor:paseo-daemon"). The supervisor then relaunches the daemon, which +# re-reads the restored config.json. This mirrors the kill in paseo-agent-shim +# and docs/decisions/supervisor-owns-daemon-wrapper-pins-worktrees.md. +GUEST_BOUNCE=' + set -eu + for pid in $(pgrep -u "$(id -u)" -f "paseo daemon start" 2>/dev/null); do + if tr "\0" " " < "/proc/$pid/cmdline" 2>/dev/null | grep -q "supervisor:paseo-daemon"; then + continue + fi + kill "$pid" 2>/dev/null || true + done + echo "paseo-restore: signaled daemon restart; the supervisor will respawn it" +' + +if [[ "${APPLY}" -ne 1 ]]; then + cat <&2 + exit 1 +fi + +# Bounce the daemon so the restored config/projects take effect. +if ! acq exec "${SANDBOX}" -- sh -c "${GUEST_BOUNCE}"; then + echo "Warning: could not signal a daemon restart. The restored state is in place;" >&2 + echo " restart the daemon manually (or recreate the sandbox) to load it." >&2 +fi + +cat < +once, so worktrees.root is re-pinned to this sandbox's project mount. As long as +the project is mounted at the same guest path, your worktrees line up with the +restored records. +EOF diff --git a/integrations/isolation/acq-kits/paseo/scripts/verify b/integrations/isolation/acq-kits/paseo/scripts/verify index e8591d99..660b3f87 100644 --- a/integrations/isolation/acq-kits/paseo/scripts/verify +++ b/integrations/isolation/acq-kits/paseo/scripts/verify @@ -184,6 +184,59 @@ else bad "paseo-register-mounts.mjs failed node --check" fi +# --- Offline check of the backup/restore host scripts (no sandbox needed). ----- +# Assert they parse, are executable, and (with 'acq' stubbed out) behave in +# dry-run by default — printing "[dry-run]" and writing/mutating NOTHING — plus +# the newest-backup auto-selection and the sandbox-name charset guard. This +# exercises the host-side logic without a live sandbox; the live round-trip +# (backup -> clear -> restore) runs later under RUN_ACQ. +info "1d. backup/restore host scripts (offline)" +BACKUP_SH="$KIT_DIR/scripts/paseo-backup" +RESTORE_SH="$KIT_DIR/scripts/paseo-restore" +for _s in "$BACKUP_SH" "$RESTORE_SH"; do + if [ ! -f "$_s" ]; then + bad "$(basename "$_s") missing at $_s" + elif [ ! -x "$_s" ]; then + bad "$(basename "$_s") is not executable" + elif bash -n "$_s" >/dev/null 2>&1; then + ok "$(basename "$_s") parses (bash -n)" + else + bad "$(basename "$_s") failed bash -n" + fi +done +if [ -x "$BACKUP_SH" ] && [ -x "$RESTORE_SH" ]; then + # Stub 'acq' so the scripts pass their CLI-present guard without a real sandbox. + _stubdir="$WORK/stub-bin"; mkdir -p "$_stubdir" + printf '#!/bin/sh\nexit 0\n' > "$_stubdir/acq"; chmod +x "$_stubdir/acq" + _hostbackups="$WORK/host-backups" + # Dry-run backup must announce dry-run and write no file. + _bout="$(PATH="$_stubdir:$PATH" HOME="$WORK/fakehome" "$BACKUP_SH" verify-sbx "$_hostbackups" 2>&1 || true)" + if printf '%s' "$_bout" | grep -q '\[dry-run\]' && [ ! -d "$_hostbackups" ]; then + ok "paseo-backup defaults to dry-run and writes nothing" + else + bad "paseo-backup dry-run did not behave (out='$_bout', dir-exists=$([ -d "$_hostbackups" ] && echo yes || echo no))" + fi + # Sandbox-name charset guard (SI-10): a bad name must be rejected non-zero. + if PATH="$_stubdir:$PATH" "$BACKUP_SH" 'bad name!' >/dev/null 2>&1; then + bad "paseo-backup accepted an unsafe sandbox name" + else + ok "paseo-backup rejects an unsafe sandbox name" + fi + # Restore auto-selects the NEWEST backup for a sandbox by lexical (=time) order. + # The script reads ~/.acq-paseo-backups, so seed that under a fake HOME. The + # seeds must be VALID (if empty) tar archives — the restore validity-checks the + # resolved tar before printing its dry-run plan. + mkdir -p "$WORK/fakehome/.acq-paseo-backups" + tar -cf "$WORK/fakehome/.acq-paseo-backups/paseo-state-verify-sbx-20200101T000000Z.tar" -T /dev/null 2>/dev/null + tar -cf "$WORK/fakehome/.acq-paseo-backups/paseo-state-verify-sbx-20990101T000000Z.tar" -T /dev/null 2>/dev/null + _rout="$(PATH="$_stubdir:$PATH" HOME="$WORK/fakehome" "$RESTORE_SH" verify-sbx 2>&1 || true)" + if printf '%s' "$_rout" | grep -q '20990101T000000Z' && printf '%s' "$_rout" | grep -q '\[dry-run\]'; then + ok "paseo-restore auto-selects the newest backup and defaults to dry-run" + else + bad "paseo-restore newest-select/dry-run did not behave (out='$_rout')" + fi +fi + # --- Select the live driver. ------------------------------------------------- DRIVER="" if [ "$RUN_ACQ" = "1" ]; then @@ -818,7 +871,73 @@ else fi fi +info "11. Backup/restore round-trip (live, via $DRIVER)" +# Prove the host-side backup/restore scripts survive a real teardown-shaped +# scenario against THIS sandbox: back up $PASEO_HOME to the host, drop a sentinel +# into the guest's $PASEO_HOME and remove it (simulating a fresh, stateless +# sandbox where the sentinel is gone), then restore and confirm the sentinel +# comes back — i.e. the snapshot round-trips through the host tar and the guest +# extract. paseo.pid must NOT appear in the archive. +if [ -z "$daemon_up" ]; then + skip "backup/restore round-trip (daemon not up, so no $PASEO_HOME to snapshot)" +elif [ ! -x "$KIT_DIR/scripts/paseo-backup" ] || [ ! -x "$KIT_DIR/scripts/paseo-restore" ]; then + skip "backup/restore round-trip (scripts missing/not executable)" +else + # Guest paths. Resolve $PASEO_HOME the same way the scripts do. + _guest_ph="$(in_sbx 'printf %s "${PASEO_HOME:-$HOME/.paseo}"')" + _sentinel="verify-roundtrip-$$-$(date -u +%s)" + # Seed a sentinel file + a fake paseo.pid so we can assert both round-trip and + # exclusion. Fail-open: if we can't seed, skip rather than falsely fail. + _seeded="$(in_sbx "mkdir -p \"$_guest_ph\" && printf '%s' '$_sentinel' > \"$_guest_ph/$_sentinel\" && printf 'stale' > \"$_guest_ph/paseo.pid\" && echo seeded")" + if [ "$_seeded" != "seeded" ]; then + skip "backup/restore round-trip (could not seed guest \$PASEO_HOME at $_guest_ph)" + else + _bkdir="$WORK/roundtrip-backups" + if ! "$KIT_DIR/scripts/paseo-backup" --apply "$SBX_NAME" "$_bkdir" >/dev/null 2>&1; then + bad "paseo-backup --apply failed against the live sandbox" + else + _tar="$(ls -1 "$_bkdir"/paseo-state-"$SBX_NAME"-*.tar 2>/dev/null | LC_ALL=C sort | tail -n 1)" + if [ -z "$_tar" ] || [ ! -s "$_tar" ]; then + bad "paseo-backup produced no non-empty tar in $_bkdir" + elif tar -tf "$_tar" 2>/dev/null | grep -qx "./paseo.pid"; then + bad "backup archive wrongly INCLUDES paseo.pid (should be excluded)" + else + ok "paseo-backup produced a valid tar excluding paseo.pid" + # Remove the sentinel from the guest to simulate lost state, then restore. + in_sbx "rm -f \"$_guest_ph/$_sentinel\"" >/dev/null + _gone="$(in_sbx "[ -e \"$_guest_ph/$_sentinel\" ] && echo present || echo gone")" + if [ "$_gone" != "gone" ]; then + bad "could not clear the sentinel before restore (round-trip inconclusive)" + elif ! "$KIT_DIR/scripts/paseo-restore" --apply "$SBX_NAME" "$_tar" >/dev/null 2>&1; then + bad "paseo-restore --apply failed against the live sandbox" + else + _back="$(in_sbx "cat \"$_guest_ph/$_sentinel\" 2>/dev/null")" + if [ "$_back" = "$_sentinel" ]; then + ok "restore round-tripped the sentinel back into \$PASEO_HOME" + else + bad "sentinel not restored (got '$_back', expected '$_sentinel')" + fi + # The restore bounces the daemon; confirm it heals back up (bounded). + _rt_deadline=$(( $(date +%s) + 45 )) + _healed="" + while :; do + [ "$(in_sbx "$health_probe")" = "yes" ] && { _healed=yes; break; } + [ "$(date +%s)" -ge "$_rt_deadline" ] && break + sleep "$POLL_INTERVAL" + done + if [ -n "$_healed" ]; then + ok "daemon healed after the restore-triggered bounce" + else + bad "daemon did not come back within 45s after the restore bounce" + fi + fi + fi + fi + # Tidy the guest sentinel + fake pid regardless of outcome (fail-open). + in_sbx "rm -f \"$_guest_ph/$_sentinel\" \"$_guest_ph/paseo.pid\"" >/dev/null 2>&1 || true + fi +fi + info "Summary" [ "$fail" -eq 0 ] && printf ' \033[32mAll checks passed.\033[0m\n' || printf ' \033[31mSome checks failed — see notes above.\033[0m\n' - [ "$fail" -eq 0 ] From fa538d655eed23baa5831c20e937deb0db697c48 Mon Sep 17 00:00:00 2001 From: Bret Mogilefsky Date: Thu, 17 Sep 2026 06:57:57 +0000 Subject: [PATCH 2/4] fix(paseo): make backup/restore work over real acq exec Live testing on an acq (msb) sandbox surfaced four issues in the initial backup/restore scripts; fix all so the round-trip actually works end-to-end: - Transport as base64 TEXT, not raw binary. acq exec hangs on a raw binary stdin/stdout stream (guest never sees EOF); text reaches EOF normally. The guest encodes on backup and the host encodes on restore, keeping the payload in acq's safe text channel on both the msb and sbx backends (no reliance on msb-specific --stream). - Snapshot-in-guest then classify tar's exit code. The daemon writes under $PASEO_HOME while tar reads it, so GNU tar exits 1/2 ("file changed as we read it") while still producing a usable archive; only rc>=3 is a real failure. - Clear the stale PID lock on the restore daemon bounce. Killing the daemon leaves paseo.pid behind, which Paseo only treats as stale after 5 minutes, so the 5s supervisor respawn loops forever on "Failed to acquire PID lock". The bounce now waits for exit and removes paseo.pid so the relaunch is clean. - Exclude re-downloadable/identity files so the archive stays small: models/ (a ~460MB speech-model cache that made base64 crawl), server-id, plus the existing daemon.log / paseo.pid / daemon-keypair.json / cli-client-id. - Also fix a BSD (macOS) mktemp incompatibility: templates must end in the XXXXXX run, so the host-side temp names were reordered. Verified live: backup produces a small valid tar with the right exclusions, restore round-trips a sentinel back into $PASEO_HOME, and the daemon returns HEALTH-OK after the bounce. Offline verify gate and unsafe-shell scan pass. Co-authored-by: OpenCode Agent --- .../isolation/acq-kits/paseo/README.md | 11 +- .../decisions/backup-restore-paseo-state.md | 113 +++++++++++-- .../acq-kits/paseo/scripts/paseo-backup | 159 ++++++++++++++---- .../acq-kits/paseo/scripts/paseo-restore | 81 +++++++-- 4 files changed, 297 insertions(+), 67 deletions(-) diff --git a/integrations/isolation/acq-kits/paseo/README.md b/integrations/isolation/acq-kits/paseo/README.md index d5f4a9bb..589aa5e6 100644 --- a/integrations/isolation/acq-kits/paseo/README.md +++ b/integrations/isolation/acq-kits/paseo/README.md @@ -173,9 +173,14 @@ scripts/paseo-restore --apply daemon so it re-reads the restored config and re-recognizes your projects and worktrees. -- **What's captured:** the full `$PASEO_HOME` tree **minus `paseo.pid`** (a - supervisor PID lock the daemon recreates on boot). Logs under - `~/.local/state/paseo` are regenerated every boot and are not captured. +- **What's captured:** the `$PASEO_HOME` tree (`config.json`, `projects/`, + `schedules/`, session records) minus files that should not travel between + sandboxes: `paseo.pid` (a PID lock the daemon recreates), `daemon.log` (the + live, multi-MB log — excluded so a concurrent write can't fail the archive), + the per-install identity `daemon-keypair.json` / `cli-client-id` / `server-id` + (regenerated on boot), and `models/` (large, re-downloadable model caches — a + speech model alone is ~460 MB). Logs under `~/.local/state/paseo` are also + regenerated every boot and are not captured. - **Worktrees:** their contents are **not** in the backup — they already persist on the host mount. The restored `$PASEO_HOME` is what makes Paseo re-recognize them, as long as the project is re-mounted at the **same guest path** (acq's diff --git a/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md b/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md index 6699e954..a9ff3578 100644 --- a/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md +++ b/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md @@ -33,32 +33,92 @@ We evaluated three durable-storage approaches before this one: **Ship two host-side helper scripts — `scripts/paseo-backup` and `scripts/paseo-restore` — that snapshot `$PASEO_HOME` to a host tar before teardown and replay it into a new sandbox afterward.** They use only `acq exec` -stdin/stdout streaming (already the kit's universal in-guest mechanism); no new -`acq` capability, no persistent-storage primitive, no coupling of Paseo state to -a project repo, and no extra mount argument. - -- **`paseo-backup [--apply] [dest-dir]`** streams - `tar -C "${PASEO_HOME:-$HOME/.paseo}" --exclude ./paseo.pid -cf - .` over - `acq exec` stdout into `/paseo-state--.tar` - (default dest `~/.acq-paseo-backups`). It writes to a temp file and atomically - renames on success, and verifies the captured stream is a non-empty, valid tar - before promoting it — an interrupted capture never leaves a truncated archive. -- **`paseo-restore [--apply] [backup-tar]`** streams a backup tar into - `${PASEO_HOME:-$HOME/.paseo}` in the target sandbox, then bounces the daemon so - it re-reads the restored config/projects. With no `backup-tar` it auto-selects - the most recent backup for that sandbox. +(already the kit's universal in-guest mechanism); no new `acq` capability, no +persistent-storage primitive, no coupling of Paseo state to a project repo, and +no extra mount argument. + +- **`paseo-backup [--apply] [dest-dir]`** tars + `${PASEO_HOME:-$HOME/.paseo}` (minus the exclusions below) inside the guest and + emits it base64-encoded over `acq exec` stdout; the host decodes it into + `/paseo-state--.tar` (default dest `~/.acq-paseo-backups`). + It writes to a temp file and atomically renames on success, and verifies the + decoded result is a valid tar before promoting it — an interrupted capture + never leaves a truncated archive. +- **`paseo-restore [--apply] [backup-tar]`** base64-encodes the backup + tar on the host, pipes the text into the guest which decodes and extracts it + into `${PASEO_HOME:-$HOME/.paseo}`, then bounces the daemon so it re-reads the + restored config/projects. With no `backup-tar` it auto-selects the most recent + backup for that sandbox. - Both **default to dry-run**; `--apply` is required to write or mutate anything. +## Transport is base64 text, not raw binary + +`acq` must work over **both** isolation backends (msb and sbx). Streaming a raw +binary tar over `acq exec` stdin/stdout was observed to **hang** on msb: a guest +`cat | wc -c` fed the tar bytes never saw EOF, so `tar -xf -` blocked forever. +Plain multi-line **text** stdin, by contrast, reaches EOF normally (verified). +`msb exec` exposes a `--stream` flag for byte-faithful binary I/O, but relying on +it would be backend-specific. + +So the archive is transported as **base64 text** in both directions — the guest +encodes on backup, the host encodes on restore — keeping the payload in the safe +text channel that behaves identically on every backend. `base64` is required on +both the host and in the guest image (present on the opencode base image). The +cost is ~33% transport inflation, which is immaterial once the snapshot is kept +small (see exclusions). This is why excluding the model cache matters: base64 of +a ~460MB model cache is ~640MB of text piped through `acq exec`, which crawled to +an apparent hang — trimming the snapshot to real state keeps the text channel +fast. + +## Capture is race-tolerant: snapshot-in-guest, then classify the tar exit code + +`acq exec` **faithfully propagates the guest command's exit code** (verified: +`exit 7` in the guest surfaces as host exit 7). That is a feature — it lets the +scripts detect real failures — but it means a naive `tar -cf - .` fails +on an *expected, non-fatal* condition: the daemon writes under `$PASEO_HOME` +while tar reads it, and GNU tar exits **1** ("file changed as we read it"), or +**2** when a whole entry is affected, *while still producing a valid archive of +everything else*. The first live `--apply` failed with exactly this (`rc=2`). + +The capture therefore does two things in the guest: + +1. **Snapshots to a guest temp file first**, then `cat`s that settled file to + stdout — so the bytes streamed to the host can't race the daemon. +2. **Classifies tar's exit code**: `0` = clean; `1`/`2` = files + changed/vanished mid-read (expected here — the archive is usable); `>= 3` = + a genuine failure (missing dir, write error), which aborts. `daemon.log` (the + busiest file) is excluded outright, and `--warning=no-file-changed + --ignore-failed-read` quiet the remaining churn. + +Because the guest capture now exits non-zero **only** on a real failure, the host +side *can* trust the propagated exit code, and additionally verifies the result +is a non-empty, valid tar (`tar -tf`) before atomically renaming it into place. +`paseo-restore` likewise checks both the propagated exit code and a completion +marker the guest prints, so a partial extract is never mistaken for success. + ## What the snapshot includes and excludes -- **Included:** the full `$PASEO_HOME` tree (`config.json`, `projects/`, - session/agent records, anything else Paseo persists there). Capturing the whole - tree — rather than an allowlist of known files — means a future Paseo state file - is preserved automatically. +- **Included:** the `$PASEO_HOME` tree (`config.json`, `projects/`, `agents/`, + session records, anything else Paseo persists there) minus the specific + exclusions below. Capturing the tree broadly — rather than an allowlist of + known files — means a future Paseo state file is preserved automatically. - **Excluded — `paseo.pid`:** a supervisor PID lock. Replaying a stale PID into a fresh container is misleading; the daemon recreates it on boot. (Paseo's lock is stale-tolerant, so a leftover would be reclaimed anyway, but there is no reason to carry it.) +- **Excluded — `daemon.log`:** the daemon's live log, observed at ~1.8 MB and + growing on a real sandbox. Because it is appended *while the tar reads it*, GNU + tar reports "file changed as we read it" and exits non-zero (rc 1, or rc 2 when + a whole entry is affected) — the observed cause of the first live backup + failing. It is pure runtime noise, not restorable state, so it is excluded. +- **Excluded — `daemon-keypair.json`, `cli-client-id`, `server-id`:** + per-daemon/-install identity that the daemon regenerates on first boot. + Carrying it into a *different* sandbox is pointless at best and confusing at + worst. +- **Excluded — `models/`:** downloaded model caches (e.g. a ~460 MB sherpa-onnx + speech model observed on a live sandbox). Large, re-downloadable, and not + state; including it both bloats the archive and makes the base64 transport + crawl. The daemon re-fetches on demand. - **Not captured — `~/.local/state/paseo`:** logs, install log, CA bundle. All regenerated every boot. - **Not captured — git worktree contents.** See below. @@ -82,6 +142,23 @@ of them (its session/worktree records under `$PASEO_HOME`). Restoring the worktrees. This keeps the design simple and avoids leaving either the host or Paseo in an inconsistent state. +## The restore daemon bounce must clear the stale PID lock + +After extracting the snapshot, `paseo-restore` restarts the daemon so it re-reads +the restored `config.json` and projects. The kit runs the daemon under a +`supervisor:paseo-daemon` respawn loop, so the bounce **kills the daemon child** +(never the supervisor) and lets the supervisor relaunch it. + +But killing the daemon leaves its `$PASEO_HOME/paseo.pid` lock behind, and +Paseo only treats that lock as stale after **five minutes**. The supervisor +respawns every ~5 s, so without intervention every relaunch loses the race — +`Failed to acquire PID lock due to race condition` on repeat, and the daemon +never comes back (observed live). The bounce therefore **kills the daemon, waits +for it to actually exit, then removes `paseo.pid`** so the next respawn acquires +the lock cleanly. The lock is meaningless once its owner is dead, so removing it +is safe. (This is also why `paseo.pid` is excluded from the snapshot — a restored +stale lock would reintroduce the same stall.) + ## `worktrees.root` reconciliation on restore A restored `config.json` carries the **old** sandbox's absolute `worktrees.root`. diff --git a/integrations/isolation/acq-kits/paseo/scripts/paseo-backup b/integrations/isolation/acq-kits/paseo/scripts/paseo-backup index a9dc267d..306581d9 100755 --- a/integrations/isolation/acq-kits/paseo/scripts/paseo-backup +++ b/integrations/isolation/acq-kits/paseo/scripts/paseo-backup @@ -11,18 +11,26 @@ # `paseo-restore` replays it into a new sandbox afterward. # # WHAT IS AND IS NOT CAPTURED: -# - Captured: the full $PASEO_HOME tree (config.json, projects/, session/agent -# records, any other state Paseo persists there) MINUS paseo.pid. -# - Excluded: paseo.pid — a supervisor PID lock. Replaying a stale PID into a -# fresh container would be misleading; the daemon recreates it on boot. +# - Captured: the $PASEO_HOME tree (config.json, projects/, agents/, session +# records, and any other state Paseo persists there) MINUS the exclusions +# below. +# - Excluded: +# * paseo.pid — supervisor PID lock; the daemon recreates it on boot. +# * daemon.log — the daemon's live, multi-MB log. It is actively appended +# while we read (which makes GNU tar exit non-zero) and is pure runtime +# noise, not state worth restoring. +# * daemon-keypair.json, cli-client-id, server-id — per-daemon/-install +# identity the daemon regenerates on first boot; carrying it into a +# DIFFERENT sandbox is pointless at best. +# * models/ — downloaded model caches (e.g. a ~460MB speech model). Large, +# re-downloadable, and not state; excluding it keeps the archive small. # - NOT captured: git worktree CONTENTS. The kit pins Paseo's worktrees under # /.paseo-worktrees, i.e. inside a host bind mount, so the # worktree files ALREADY survive recreation on their own (as long as the # project is re-mounted at the same guest path). What is lost without this # backup is Paseo's KNOWLEDGE of them (its session/worktree records under # $PASEO_HOME) — restoring this snapshot is exactly what makes Paseo -# re-recognize the on-disk worktrees. Logs under ~/.local/state/paseo are -# regenerated every boot and are also not captured. +# re-recognize the on-disk worktrees. # See docs/decisions/backup-restore-paseo-state.md. # # FILENAME CONTRACT (consumed by paseo-restore's newest-backup default): @@ -50,13 +58,16 @@ SANDBOX="" DEST_DIR="" DEFAULT_DEST="${HOME}/.acq-paseo-backups" TMP_OUT="" +TMP_ERR="" +TMP_B64="" usage() { cat <<'EOF' Usage: paseo-backup [--apply] [dest-dir] -Snapshot a running acq sandbox's Paseo state ($PASEO_HOME, minus paseo.pid) to a -host tar so it survives sandbox recreation. +Snapshot a running acq sandbox's Paseo state ($PASEO_HOME, minus its PID lock, +live log, and per-install identity files) to a host tar so it survives sandbox +recreation. Arguments: Name of the running acq sandbox to back up. @@ -134,10 +145,38 @@ STAMP="$(date -u +%Y%m%dT%H%M%SZ)" OUT_FILE="${DEST_DIR}/paseo-state-${SANDBOX}-${STAMP}.tar" # The guest-side capture. Resolve $PASEO_HOME exactly like the kit's own helpers -# (${PASEO_HOME:-$HOME/.paseo}) and stream a tar of its contents (minus the PID -# lock) to stdout. Read-only against the sandbox. `--exclude ./paseo.pid` is -# relative to the -C dir. If $PASEO_HOME does not exist yet, tar has nothing to -# archive and we detect the empty result below rather than write a bogus file. +# (${PASEO_HOME:-$HOME/.paseo}) and emit a base64-encoded tar of its contents to +# stdout. Read-only against the sandbox. +# +# WHY THIS IS NOT A PLAIN `tar -cf - .`: two independent problems, both fixed +# here so the capture is backend-agnostic (works over msb AND sbx via acq). +# +# 1. LIVE-FILE CHURN. The daemon writes under $PASEO_HOME while we read it, and +# GNU tar treats a file changing mid-read as an error — exit 1 ("file +# changed as we read it") or 2 for a whole entry — WHILE STILL emitting a +# valid archive of everything else. We observed rc=2 on a live sandbox. +# Fix: snapshot to a guest TEMP FILE with churn tolerated +# (--warning=no-file-changed, --ignore-failed-read), and classify tar's exit +# code (0 clean; 1/2 expected churn; >=3 real failure → abort). +# +# 2. BINARY STDIN/STDOUT OVER acq. `acq exec` was observed to hang on a RAW +# BINARY stream (a guest `cat | wc -c` never saw EOF), while multi-line TEXT +# reaches EOF normally. So we do NOT stream raw tar bytes: the guest +# base64-ENCODES the archive to stdout, and the host base64-DECODES it. The +# restore side mirrors this (host encodes, guest decodes). base64 keeps the +# payload in acq's safe text channel on both backends. +# +# EXCLUSIONS (paths are relative to the -C dir, i.e. $PASEO_HOME root): +# - paseo.pid — supervisor PID lock; recreated on boot. +# - daemon.log — the daemon's live, multi-MB log; pure runtime noise and +# the most churn-prone file. +# - daemon-keypair.json, cli-client-id, server-id — per-daemon/-install +# identity the daemon regenerates on first boot; carrying +# it into a DIFFERENT sandbox is pointless at best. +# - models/ — downloaded model caches (e.g. the ~460MB sherpa-onnx +# speech model). Large, re-downloadable, NOT state. Leaving +# it in bloats the archive to hundreds of MB and makes the +# base64 transport crawl; the daemon re-fetches on demand. GUEST_CAPTURE=' set -eu PH="${PASEO_HOME:-$HOME/.paseo}" @@ -145,20 +184,45 @@ GUEST_CAPTURE=' echo "paseo-backup: $PH does not exist in the sandbox" >&2 exit 3 fi - tar -C "$PH" --exclude ./paseo.pid -cf - . + TMP="$(mktemp "${TMPDIR:-/tmp}/paseo-backup.XXXXXX.tar")" + trap "rm -f \"$TMP\"" EXIT INT TERM + rc=0 + tar -C "$PH" \ + --warning=no-file-changed \ + --ignore-failed-read \ + --exclude ./paseo.pid \ + --exclude ./daemon.log \ + --exclude ./daemon-keypair.json \ + --exclude ./cli-client-id \ + --exclude ./server-id \ + --exclude ./models \ + -cf "$TMP" . || rc=$? + # rc 0 = clean; 1/2 = files changed/vanished during read (expected here, the + # archive is still usable); >=3 = fatal — abort. + if [ "$rc" -ge 3 ]; then + echo "paseo-backup: tar failed with exit $rc" >&2 + exit "$rc" + fi + base64 < "$TMP" ' cleanup() { if [[ -n "${TMP_OUT}" && -f "${TMP_OUT}" ]]; then rm -f "${TMP_OUT}" fi + if [[ -n "${TMP_ERR}" && -f "${TMP_ERR}" ]]; then + rm -f "${TMP_ERR}" + fi + if [[ -n "${TMP_B64}" && -f "${TMP_B64}" ]]; then + rm -f "${TMP_B64}" + fi } trap cleanup EXIT INT TERM if [[ "${APPLY}" -ne 1 ]]; then cat <"${TMP_OUT}"; then - echo "Error: 'acq exec ${SANDBOX}' failed — is the sandbox running?" >&2 +if ! command -v base64 >/dev/null 2>&1; then + echo "Error: 'base64' is required on the host to decode the archive." >&2 + exit 2 +fi + +# The guest emits the archive as base64 TEXT (raw binary over acq stdin/stdout +# hangs — see the GUEST_CAPTURE header). Capture that text to a temp file, then +# decode it into the final .tar and atomically rename on success — an interrupted +# or failed capture never leaves a truncated .tar that paseo-restore might later +# pick as "most recent". +TMP_B64="$(mktemp "${DEST_DIR}/.paseo-state-${SANDBOX}.b64.XXXXXX")" +TMP_OUT="$(mktemp "${DEST_DIR}/.paseo-state-${SANDBOX}.tar.XXXXXX")" +# Capture the guest's stderr separately so we can surface the ACTUAL cause (e.g. +# "$PASEO_HOME does not exist", an acq error, a permission problem) rather than a +# generic guess. +TMP_ERR="$(mktemp "${DEST_DIR}/.paseo-state-${SANDBOX}.err.XXXXXX")" + +# Run the capture. acq faithfully propagates the guest exit code (verified), and +# the guest capture above exits 0 for the expected file-churn conditions and +# non-zero only for a real failure — so we CAN branch on the exit code. Redirect +# stdin from /dev/null so acq never blocks on a TTY; keep guest stderr for +# diagnostics. +acq_rc=0 +acq exec "${SANDBOX}" -- sh -c "${GUEST_CAPTURE}" \ + "${TMP_B64}" 2>"${TMP_ERR}" || acq_rc=$? + +if [[ "${acq_rc}" -ne 0 ]]; then + echo "Error: backing up '${SANDBOX}' failed (exit ${acq_rc}). The sandbox / acq reported:" >&2 + if [[ -s "${TMP_ERR}" ]]; then + sed 's/^/ | /' "${TMP_ERR}" >&2 + else + echo " | (no error output — is 'acq' on PATH and the sandbox running?)" >&2 + fi exit 1 fi -if [[ ! -s "${TMP_OUT}" ]]; then - echo "Error: captured an empty archive — the sandbox may not be running or" >&2 - echo " \$PASEO_HOME is absent. Nothing was written." >&2 +if [[ ! -s "${TMP_B64}" ]]; then + echo "Error: captured an empty archive from '${SANDBOX}'. Nothing was written." >&2 + if [[ -s "${TMP_ERR}" ]]; then + echo "The sandbox / acq reported:" >&2 + sed 's/^/ | /' "${TMP_ERR}" >&2 + else + echo " (\$PASEO_HOME may be absent, or the sandbox may not be running.)" >&2 + fi + exit 1 +fi + +# Decode the base64 text into the real tar. +if ! base64 -d <"${TMP_B64}" >"${TMP_OUT}" 2>/dev/null; then + echo "Error: could not base64-decode the captured stream; refusing to keep it." >&2 exit 1 fi # Sanity-check that we actually got a readable tar before promoting it. if ! tar -tf "${TMP_OUT}" >/dev/null 2>&1; then - echo "Error: captured data is not a valid tar archive; refusing to keep it." >&2 + echo "Error: decoded data is not a valid tar archive; refusing to keep it." >&2 + if [[ -s "${TMP_ERR}" ]]; then + echo "The sandbox / acq reported:" >&2 + sed 's/^/ | /' "${TMP_ERR}" >&2 + fi exit 1 fi diff --git a/integrations/isolation/acq-kits/paseo/scripts/paseo-restore b/integrations/isolation/acq-kits/paseo/scripts/paseo-restore index 31c91e3d..e920b67c 100755 --- a/integrations/isolation/acq-kits/paseo/scripts/paseo-restore +++ b/integrations/isolation/acq-kits/paseo/scripts/paseo-restore @@ -158,31 +158,57 @@ if ! tar -tf "${BACKUP_TAR}" >/dev/null 2>&1; then exit 1 fi -# The guest-side restore. Recreate $PASEO_HOME if needed and extract the archive -# into it, then the caller bounces the daemon separately (below). Resolves -# $PASEO_HOME the same way the kit's helpers do. +# The guest-side restore. Recreate $PASEO_HOME if needed, then base64-DECODE the +# archive from stdin and extract it. We transport the tar as BASE64 TEXT, never +# raw binary: `acq exec` was observed to hang on a binary stdin stream (a guest +# `cat | wc -c` never saw EOF), while multi-line TEXT stdin reaches EOF normally. +# base64 keeps the payload in the safe text channel. Prints a unique DONE marker +# on a completed extract; acq faithfully propagates the guest exit code. GUEST_RESTORE=' set -eu PH="${PASEO_HOME:-$HOME/.paseo}" mkdir -p "$PH" - tar -C "$PH" -xf - - echo "paseo-restore: extracted into $PH" + base64 -d | tar -C "$PH" -xf - + echo "PASEO_RESTORE_EXTRACT_OK $PH" ' -# The daemon bounce: kill only the Paseo-owned `paseo daemon start` process tree, -# NEVER the kit's supervisor respawn loop (whose argv carries the marker -# "supervisor:paseo-daemon"). The supervisor then relaunches the daemon, which -# re-reads the restored config.json. This mirrors the kill in paseo-agent-shim -# and docs/decisions/supervisor-owns-daemon-wrapper-pins-worktrees.md. +# The daemon bounce: kill the Paseo-owned `paseo daemon start` process tree (NEVER +# the kit's supervisor respawn loop, whose argv carries the marker +# "supervisor:paseo-daemon"), then REMOVE the stale PID lock so the supervisor's +# next relaunch can acquire it cleanly. +# +# Why the lock removal is required: killing the daemon leaves its `paseo.pid` +# behind, and Paseo's lock is only treated as stale after >5 minutes. The +# supervisor respawns every 5s, so without clearing the lock the fresh daemon +# loses the race every time — "Failed to acquire PID lock due to race condition" +# on repeat, and the daemon never comes back (observed live). We hold the kill → +# confirm the process is gone → delete paseo.pid, so the relaunch starts clean. +# The lock is meaningless once its owner is dead, so removing it is safe. GUEST_BOUNCE=' set -eu + PH="${PASEO_HOME:-$HOME/.paseo}" + # Kill the daemon tree, but never the supervisor loop. for pid in $(pgrep -u "$(id -u)" -f "paseo daemon start" 2>/dev/null); do if tr "\0" " " < "/proc/$pid/cmdline" 2>/dev/null | grep -q "supervisor:paseo-daemon"; then continue fi kill "$pid" 2>/dev/null || true done - echo "paseo-restore: signaled daemon restart; the supervisor will respawn it" + # Wait briefly for the daemon (non-supervisor) to actually exit before clearing + # the lock, so we do not race a still-shutting-down daemon. + i=0 + while [ "$i" -lt 20 ]; do + still="" + for pid in $(pgrep -u "$(id -u)" -f "paseo daemon start" 2>/dev/null); do + tr "\0" " " < "/proc/$pid/cmdline" 2>/dev/null | grep -q "supervisor:paseo-daemon" && continue + still=1 + done + [ -z "$still" ] && break + i=$((i + 1)) + sleep 0.25 + done + rm -f "$PH/paseo.pid" + echo "paseo-restore: killed daemon + cleared PID lock; the supervisor will respawn it" ' if [[ "${APPLY}" -ne 1 ]]; then @@ -198,17 +224,38 @@ EOF exit 0 fi -# Stream the archive into the guest. -if ! acq exec "${SANDBOX}" -- sh -c "${GUEST_RESTORE}" <"${BACKUP_TAR}"; then - echo "Error: 'acq exec ${SANDBOX}' failed during extract — is the sandbox running?" >&2 +# Stream the archive into the guest as BASE64 TEXT. Raw binary on acq's stdin was +# observed to hang (guest never sees EOF); base64 text reaches EOF normally. We +# pipe `base64 | acq exec ... 'base64 -d | tar -x'` and write acq's output +# to a temp file (no `$(...)` around acq, which also held its channel open). acq +# faithfully propagates the guest exit code; we ALSO require the guest completion +# marker, since a partial extract is worse than a clean failure. +if ! command -v base64 >/dev/null 2>&1; then + echo "Error: 'base64' is required on the host to transport the archive." >&2 + exit 2 +fi +RESTORE_OUT_FILE="$(mktemp "${TMPDIR:-/tmp}/paseo-restore.XXXXXX")" +restore_rc=0 +base64 <"${BACKUP_TAR}" | acq exec "${SANDBOX}" -- sh -c "${GUEST_RESTORE}" \ + >"${RESTORE_OUT_FILE}" 2>&1 || restore_rc=$? +RESTORE_OUT="$(cat "${RESTORE_OUT_FILE}")" +rm -f "${RESTORE_OUT_FILE}" +if [[ "${restore_rc}" -ne 0 ]] || ! printf '%s\n' "${RESTORE_OUT}" | grep -q '^PASEO_RESTORE_EXTRACT_OK '; then + echo "Error: restore into '${SANDBOX}' did not complete (exit ${restore_rc}). The sandbox / acq reported:" >&2 + if [[ -n "${RESTORE_OUT}" ]]; then + printf '%s\n' "${RESTORE_OUT}" | sed 's/^/ | /' >&2 + else + echo " | (no output — is 'acq' on PATH and the sandbox running?)" >&2 + fi exit 1 fi -# Bounce the daemon so the restored config/projects take effect. -if ! acq exec "${SANDBOX}" -- sh -c "${GUEST_BOUNCE}"; then +# Bounce the daemon so the restored config/projects take effect. Best-effort: +# the state is already in place, so a bounce hiccup is a warning, not a failure. +acq exec "${SANDBOX}" -- sh -c "${GUEST_BOUNCE}" /dev/null 2>&1 || { echo "Warning: could not signal a daemon restart. The restored state is in place;" >&2 echo " restart the daemon manually (or recreate the sandbox) to load it." >&2 -fi +} cat < Date: Thu, 17 Sep 2026 07:02:38 +0000 Subject: [PATCH 3/4] fix(paseo): mark scripts/verify executable The verify script was tracked as 100644, so a fresh checkout had to chmod +x it before RUN_ACQ=1 would run. Set the executable bit to match paseo-backup / paseo-restore. Co-authored-by: OpenCode Agent --- integrations/isolation/acq-kits/paseo/scripts/verify | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 integrations/isolation/acq-kits/paseo/scripts/verify diff --git a/integrations/isolation/acq-kits/paseo/scripts/verify b/integrations/isolation/acq-kits/paseo/scripts/verify old mode 100644 new mode 100755 From 1295191aa75639996085691a36db858f46f43fa6 Mon Sep 17 00:00:00 2001 From: Bret Mogilefsky Date: Sat, 19 Sep 2026 21:06:08 +0000 Subject: [PATCH 4/4] fix(paseo): prune logs during state restore Exclude rotated daemon logs from new backups, filter legacy backup logs before restore transport, and swap restored state into place only after a complete extract. Co-authored-by: OpenCode Agent --- .../isolation/acq-kits/paseo/README.md | 20 +++--- .../decisions/backup-restore-paseo-state.md | 19 ++--- .../acq-kits/paseo/scripts/paseo-backup | 12 ++-- .../acq-kits/paseo/scripts/paseo-restore | 71 ++++++++++++++++--- .../isolation/acq-kits/paseo/scripts/verify | 59 ++++++++++++--- 5 files changed, 141 insertions(+), 40 deletions(-) diff --git a/integrations/isolation/acq-kits/paseo/README.md b/integrations/isolation/acq-kits/paseo/README.md index 589aa5e6..f973cf79 100644 --- a/integrations/isolation/acq-kits/paseo/README.md +++ b/integrations/isolation/acq-kits/paseo/README.md @@ -169,18 +169,20 @@ scripts/paseo-restore scripts/paseo-restore --apply ``` -`paseo-restore` streams the snapshot back into `$PASEO_HOME` and restarts the -daemon so it re-reads the restored config and re-recognizes your projects and -worktrees. +`paseo-restore` prunes legacy runtime logs from older backups, streams the +snapshot back into a temporary directory in the sandbox, atomically swaps it into +`$PASEO_HOME` after a complete extract, and restarts the daemon so it re-reads +the restored config and re-recognizes your projects and worktrees. - **What's captured:** the `$PASEO_HOME` tree (`config.json`, `projects/`, `schedules/`, session records) minus files that should not travel between - sandboxes: `paseo.pid` (a PID lock the daemon recreates), `daemon.log` (the - live, multi-MB log — excluded so a concurrent write can't fail the archive), - the per-install identity `daemon-keypair.json` / `cli-client-id` / `server-id` - (regenerated on boot), and `models/` (large, re-downloadable model caches — a - speech model alone is ~460 MB). Logs under `~/.local/state/paseo` are also - regenerated every boot and are not captured. + sandboxes: `paseo.pid` (a PID lock the daemon recreates), `daemon.log`, + `daemon.log.*`, `daemon.log.txt`, and `*-daemon.log` (live or rotated runtime + logs — excluded so concurrent writes cannot fail the archive and restore does + not crawl through non-state data), the per-install identity `daemon-keypair.json` + / `cli-client-id` / `server-id` (regenerated on boot), and `models/` (large, + re-downloadable model caches — a speech model alone is ~460 MB). Logs under + `~/.local/state/paseo` are also regenerated every boot and are not captured. - **Worktrees:** their contents are **not** in the backup — they already persist on the host mount. The restored `$PASEO_HOME` is what makes Paseo re-recognize them, as long as the project is re-mounted at the **same guest path** (acq's diff --git a/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md b/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md index a9ff3578..ee7d6c13 100644 --- a/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md +++ b/integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md @@ -44,11 +44,12 @@ no extra mount argument. It writes to a temp file and atomically renames on success, and verifies the decoded result is a valid tar before promoting it — an interrupted capture never leaves a truncated archive. -- **`paseo-restore [--apply] [backup-tar]`** base64-encodes the backup - tar on the host, pipes the text into the guest which decodes and extracts it - into `${PASEO_HOME:-$HOME/.paseo}`, then bounces the daemon so it re-reads the - restored config/projects. With no `backup-tar` it auto-selects the most recent - backup for that sandbox. +- **`paseo-restore [--apply] [backup-tar]`** prunes legacy runtime logs + from older backups, base64-encodes the backup tar on the host, pipes the text + into the guest which decodes and extracts it into a temporary sibling directory, + atomically swaps that directory into `${PASEO_HOME:-$HOME/.paseo}`, then bounces + the daemon so it re-reads the restored config/projects. With no `backup-tar` it + auto-selects the most recent backup for that sandbox. - Both **default to dry-run**; `--apply` is required to write or mutate anything. ## Transport is base64 text, not raw binary @@ -106,11 +107,13 @@ marker the guest prints, so a partial extract is never mistaken for success. fresh container is misleading; the daemon recreates it on boot. (Paseo's lock is stale-tolerant, so a leftover would be reclaimed anyway, but there is no reason to carry it.) -- **Excluded — `daemon.log`:** the daemon's live log, observed at ~1.8 MB and - growing on a real sandbox. Because it is appended *while the tar reads it*, GNU +- **Excluded — `daemon.log`, `daemon.log.*`, `daemon.log.txt`, and + `*-daemon.log`:** live or rotated daemon logs can be multi-MB and are not + restorable state. Because live logs are appended *while the tar reads them*, GNU tar reports "file changed as we read it" and exits non-zero (rc 1, or rc 2 when a whole entry is affected) — the observed cause of the first live backup - failing. It is pure runtime noise, not restorable state, so it is excluded. + failing. Rotated logs have also made restore appear hung because they can be + streamed before useful state files over acq's base64 text channel. - **Excluded — `daemon-keypair.json`, `cli-client-id`, `server-id`:** per-daemon/-install identity that the daemon regenerates on first boot. Carrying it into a *different* sandbox is pointless at best and confusing at diff --git a/integrations/isolation/acq-kits/paseo/scripts/paseo-backup b/integrations/isolation/acq-kits/paseo/scripts/paseo-backup index 306581d9..276e912b 100755 --- a/integrations/isolation/acq-kits/paseo/scripts/paseo-backup +++ b/integrations/isolation/acq-kits/paseo/scripts/paseo-backup @@ -16,9 +16,8 @@ # below. # - Excluded: # * paseo.pid — supervisor PID lock; the daemon recreates it on boot. -# * daemon.log — the daemon's live, multi-MB log. It is actively appended -# while we read (which makes GNU tar exit non-zero) and is pure runtime -# noise, not state worth restoring. +# * daemon.log and rotated daemon logs — live/runtime noise that can bloat +# the archive and make restore look hung over acq's text transport. # * daemon-keypair.json, cli-client-id, server-id — per-daemon/-install # identity the daemon regenerates on first boot; carrying it into a # DIFFERENT sandbox is pointless at best. @@ -168,8 +167,8 @@ OUT_FILE="${DEST_DIR}/paseo-state-${SANDBOX}-${STAMP}.tar" # # EXCLUSIONS (paths are relative to the -C dir, i.e. $PASEO_HOME root): # - paseo.pid — supervisor PID lock; recreated on boot. -# - daemon.log — the daemon's live, multi-MB log; pure runtime noise and -# the most churn-prone file. +# - daemon.log, daemon.log.*, *-daemon.log, daemon.log.txt — live/runtime +# log artifacts; pure noise and the most churn-prone files. # - daemon-keypair.json, cli-client-id, server-id — per-daemon/-install # identity the daemon regenerates on first boot; carrying # it into a DIFFERENT sandbox is pointless at best. @@ -192,6 +191,9 @@ GUEST_CAPTURE=' --ignore-failed-read \ --exclude ./paseo.pid \ --exclude ./daemon.log \ + --exclude ./daemon.log.* \ + --exclude ./daemon.log.txt \ + --exclude ./*-daemon.log \ --exclude ./daemon-keypair.json \ --exclude ./cli-client-id \ --exclude ./server-id \ diff --git a/integrations/isolation/acq-kits/paseo/scripts/paseo-restore b/integrations/isolation/acq-kits/paseo/scripts/paseo-restore index e920b67c..cfcfb3d5 100755 --- a/integrations/isolation/acq-kits/paseo/scripts/paseo-restore +++ b/integrations/isolation/acq-kits/paseo/scripts/paseo-restore @@ -46,6 +46,8 @@ APPLY=0 SANDBOX="" BACKUP_TAR="" DEFAULT_DEST="${HOME}/.acq-paseo-backups" +TMP_FILTER_DIR="" +TMP_FILTERED_TAR="" usage() { cat <<'EOF' @@ -158,17 +160,53 @@ if ! tar -tf "${BACKUP_TAR}" >/dev/null 2>&1; then exit 1 fi -# The guest-side restore. Recreate $PASEO_HOME if needed, then base64-DECODE the -# archive from stdin and extract it. We transport the tar as BASE64 TEXT, never -# raw binary: `acq exec` was observed to hang on a binary stdin stream (a guest -# `cat | wc -c` never saw EOF), while multi-line TEXT stdin reaches EOF normally. -# base64 keeps the payload in the safe text channel. Prints a unique DONE marker -# on a completed extract; acq faithfully propagates the guest exit code. +cleanup() { + if [[ -n "${TMP_FILTER_DIR}" && -d "${TMP_FILTER_DIR}" ]]; then + rm -rf "${TMP_FILTER_DIR}" + fi + if [[ -n "${TMP_FILTERED_TAR}" && -f "${TMP_FILTERED_TAR}" ]]; then + rm -f "${TMP_FILTERED_TAR}" + fi +} +trap cleanup EXIT INT TERM + +# The guest-side restore. Decode and extract into a sibling temp directory, then +# swap it into $PASEO_HOME only after the extract is complete. We transport the +# tar as BASE64 TEXT, never raw binary: `acq exec` was observed to hang on a +# binary stdin stream (a guest `cat | wc -c` never saw EOF), while multi-line TEXT +# stdin reaches EOF normally. base64 keeps the payload in the safe text channel. +# Prints a unique DONE marker on a completed extract; acq faithfully propagates +# the guest exit code. GUEST_RESTORE=' set -eu PH="${PASEO_HOME:-$HOME/.paseo}" - mkdir -p "$PH" - base64 -d | tar -C "$PH" -xf - + parent="$(dirname "$PH")" + base="$(basename "$PH")" + mkdir -p "$parent" + tmp="$(mktemp -d "$parent/.${base}.restore.XXXXXX")" + old="" + cleanup_restore() { + [ -n "$tmp" ] && [ -d "$tmp" ] && rm -rf "$tmp" + if [ -n "$old" ] && [ -e "$old" ] && [ ! -e "$PH" ]; then + mv "$old" "$PH" 2>/dev/null || true + fi + } + trap cleanup_restore EXIT INT TERM + base64 -d | tar -C "$tmp" -xf - + if [ -e "$PH" ]; then + old="$(mktemp -d "$parent/.${base}.old.XXXXXX")" + rmdir "$old" + mv "$PH" "$old" + fi + if ! mv "$tmp" "$PH"; then + if [ -n "$old" ] && [ -e "$old" ] && [ ! -e "$PH" ]; then + mv "$old" "$PH" 2>/dev/null || true + fi + exit 1 + fi + tmp="" + [ -n "$old" ] && rm -rf "$old" + trap - EXIT INT TERM echo "PASEO_RESTORE_EXTRACT_OK $PH" ' @@ -224,6 +262,21 @@ EOF exit 0 fi +# Older backups may include rotated daemon logs. They are not state, and when +# they appear before config/projects/agents in tar order they make restore look +# hung over acq's base64 text transport. Prune them before streaming. +if tar -tf "${BACKUP_TAR}" | grep -Eq '(^|/)daemon\.log(\.|$)|(^|/)[0-9]{8}-[0-9]{4}-[0-9]{2}-daemon\.log$|(^|/)daemon\.log\.txt$'; then + TMP_FILTER_DIR="$(mktemp -d "${TMPDIR:-/tmp}/paseo-restore-filter.XXXXXX")" + TMP_FILTERED_TAR="$(mktemp "${TMPDIR:-/tmp}/paseo-restore-filtered.XXXXXX.tar")" + echo "paseo-restore: pruning runtime log files from backup before transport" >&2 + tar -xf "${BACKUP_TAR}" -C "${TMP_FILTER_DIR}" + find "${TMP_FILTER_DIR}" -type f \ + \( -name 'daemon.log' -o -name 'daemon.log.*' -o -name 'daemon.log.txt' -o -name '*-daemon.log' \) \ + -delete + tar -C "${TMP_FILTER_DIR}" -cf "${TMP_FILTERED_TAR}" . + BACKUP_TAR="${TMP_FILTERED_TAR}" +fi + # Stream the archive into the guest as BASE64 TEXT. Raw binary on acq's stdin was # observed to hang (guest never sees EOF); base64 text reaches EOF normally. We # pipe `base64 | acq exec ... 'base64 -d | tar -x'` and write acq's output @@ -235,6 +288,8 @@ if ! command -v base64 >/dev/null 2>&1; then exit 2 fi RESTORE_OUT_FILE="$(mktemp "${TMPDIR:-/tmp}/paseo-restore.XXXXXX")" +RESTORE_BYTES="$(wc -c <"${BACKUP_TAR}" | tr -d ' ')" +echo "paseo-restore: streaming ${RESTORE_BYTES} bytes into '${SANDBOX}' (waiting for completion marker)" >&2 restore_rc=0 base64 <"${BACKUP_TAR}" | acq exec "${SANDBOX}" -- sh -c "${GUEST_RESTORE}" \ >"${RESTORE_OUT_FILE}" 2>&1 || restore_rc=$? diff --git a/integrations/isolation/acq-kits/paseo/scripts/verify b/integrations/isolation/acq-kits/paseo/scripts/verify index 660b3f87..226f5d65 100755 --- a/integrations/isolation/acq-kits/paseo/scripts/verify +++ b/integrations/isolation/acq-kits/paseo/scripts/verify @@ -235,6 +235,43 @@ if [ -x "$BACKUP_SH" ] && [ -x "$RESTORE_SH" ]; then else bad "paseo-restore newest-select/dry-run did not behave (out='$_rout')" fi + + _legacy_src="$WORK/legacy-restore-src" + _legacy_out="$WORK/legacy-restore-out" + _legacy_tar="$WORK/legacy-restore.tar" + mkdir -p "$_legacy_src/projects" "$_legacy_src/agents/example" "$_legacy_out" + printf 'noise' > "$_legacy_src/20260101-0000-01-daemon.log" + printf 'noise' > "$_legacy_src/daemon.log.txt" + printf '{"ok":true}\n' > "$_legacy_src/config.json" + printf '[]\n' > "$_legacy_src/projects/projects.json" + printf '[]\n' > "$_legacy_src/projects/workspaces.json" + printf '{}\n' > "$_legacy_src/agents/example/agent.json" + tar -C "$_legacy_src" -cf "$_legacy_tar" . + cat > "$_stubdir/acq" <<'SH' +#!/bin/sh +if [ "$1" = "exec" ]; then + while [ "$#" -gt 0 ] && [ "$1" != "--" ]; do + shift + done + [ "$#" -gt 0 ] && shift + case "$*" in + *'base64 -d'*) exec "$@" ;; + *) exit 0 ;; + esac +else + exit 0 +fi +SH + chmod +x "$_stubdir/acq" + _restore_apply_out="$(PATH="$_stubdir:$PATH" HOME="$WORK/fakehome" PASEO_HOME="$_legacy_out/.paseo" "$RESTORE_SH" --apply verify-sbx "$_legacy_tar" 2>&1 || true)" + if [ -f "$_legacy_out/.paseo/config.json" ] \ + && [ ! -e "$_legacy_out/.paseo/20260101-0000-01-daemon.log" ] \ + && [ ! -e "$_legacy_out/.paseo/daemon.log.txt" ] \ + && printf '%s' "$_restore_apply_out" | grep -q 'streaming'; then + ok "paseo-restore filters legacy logs and reports progress" + else + bad "paseo-restore legacy filter/progress failed (out='$_restore_apply_out')" + fi fi # --- Select the live driver. ------------------------------------------------- @@ -883,12 +920,14 @@ if [ -z "$daemon_up" ]; then elif [ ! -x "$KIT_DIR/scripts/paseo-backup" ] || [ ! -x "$KIT_DIR/scripts/paseo-restore" ]; then skip "backup/restore round-trip (scripts missing/not executable)" else - # Guest paths. Resolve $PASEO_HOME the same way the scripts do. - _guest_ph="$(in_sbx 'printf %s "${PASEO_HOME:-$HOME/.paseo}"')" - _sentinel="verify-roundtrip-$$-$(date -u +%s)" - # Seed a sentinel file + a fake paseo.pid so we can assert both round-trip and - # exclusion. Fail-open: if we can't seed, skip rather than falsely fail. - _seeded="$(in_sbx "mkdir -p \"$_guest_ph\" && printf '%s' '$_sentinel' > \"$_guest_ph/$_sentinel\" && printf 'stale' > \"$_guest_ph/paseo.pid\" && echo seeded")" + # Guest paths. Resolve $PASEO_HOME the same way the scripts do. + _guest_ph="$(in_sbx 'printf %s "${PASEO_HOME:-$HOME/.paseo}"')" + _sentinel="verify-roundtrip-$$-$(date -u +%s)" + # Seed a sentinel file, fake paseo.pid, and rotated log artifacts so we can + # assert round-trip and exclusions. Fail-open: if we can't seed, skip rather + # than falsely fail. + _seeded="$(in_sbx "mkdir -p \"$_guest_ph\" && printf '%s' '$_sentinel' > \"$_guest_ph/$_sentinel\" && printf 'stale' > \"$_guest_ph/paseo.pid\" && printf 'noise' > \"$_guest_ph/20260101-0000-01-daemon.log\" && printf 'noise' > \"$_guest_ph/daemon.log.txt\" && echo seeded")" + if [ "$_seeded" != "seeded" ]; then skip "backup/restore round-trip (could not seed guest \$PASEO_HOME at $_guest_ph)" else @@ -899,10 +938,10 @@ else _tar="$(ls -1 "$_bkdir"/paseo-state-"$SBX_NAME"-*.tar 2>/dev/null | LC_ALL=C sort | tail -n 1)" if [ -z "$_tar" ] || [ ! -s "$_tar" ]; then bad "paseo-backup produced no non-empty tar in $_bkdir" - elif tar -tf "$_tar" 2>/dev/null | grep -qx "./paseo.pid"; then - bad "backup archive wrongly INCLUDES paseo.pid (should be excluded)" + elif tar -tf "$_tar" 2>/dev/null | grep -Eq "^\./(paseo\.pid|20260101-0000-01-daemon\.log|daemon\.log\.txt)$"; then + bad "backup archive wrongly INCLUDES runtime-only files" else - ok "paseo-backup produced a valid tar excluding paseo.pid" + ok "paseo-backup produced a valid tar excluding runtime-only files" # Remove the sentinel from the guest to simulate lost state, then restore. in_sbx "rm -f \"$_guest_ph/$_sentinel\"" >/dev/null _gone="$(in_sbx "[ -e \"$_guest_ph/$_sentinel\" ] && echo present || echo gone")" @@ -934,7 +973,7 @@ else fi fi # Tidy the guest sentinel + fake pid regardless of outcome (fail-open). - in_sbx "rm -f \"$_guest_ph/$_sentinel\" \"$_guest_ph/paseo.pid\"" >/dev/null 2>&1 || true + in_sbx "rm -f \"$_guest_ph/$_sentinel\" \"$_guest_ph/paseo.pid\" \"$_guest_ph/20260101-0000-01-daemon.log\" \"$_guest_ph/daemon.log.txt\"" >/dev/null 2>&1 || true fi fi