Skip to content

feat(paseo): add backup/restore scripts to preserve state across sandbox recreation - #420

Closed
mogul wants to merge 4 commits into
mainfrom
grey-bulldog
Closed

feat(paseo): add backup/restore scripts to preserve state across sandbox recreation#420
mogul wants to merge 4 commits into
mainfrom
grey-bulldog

Conversation

@mogul

@mogul mogul commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Recreating an acq sandbox loses all Paseo state: $PASEO_HOME (config.json,
the projects registry, session/agent records, schedules) lives on the ephemeral
guest filesystem, so acq rm + a fresh create drops it — and the on-disk git
worktrees (which persist on the host mount) show up orphaned because Paseo no
longer has their records.

This adds two host-side helper scripts that snapshot Paseo state to the host
before teardown and replay it into a new sandbox afterward, using only
acq exec (no new acq capability, no persistent-volume primitive, no coupling
of state to a project repo, no extra mount argument):

  • scripts/paseo-backup [--apply] <sandbox> [dest-dir] — snapshots
    $PASEO_HOME to ~/.acq-paseo-backups/paseo-state-<sandbox>-<UTC>.tar.
  • scripts/paseo-restore [--apply] <sandbox> [backup-tar] — replays a snapshot
    and restarts the daemon; with no tar it auto-selects the newest backup for the
    sandbox.

Both default to dry-run; --apply is required to write or mutate anything.

Why backup/restore (not a persistent volume)

Three storage approaches were considered. Relocating $PASEO_HOME onto the
project mount couples state to a repo dir; a dedicated state mount forces an
extra acq run argument; and the hybrid/v1 volumes primitive is
creation-time/unseeded/per-sandbox and cannot provide cross-recreate persistence
without an upstream acq-side change. Backup/restore needs none of that and
works today. Full rationale in
integrations/isolation/acq-kits/paseo/docs/decisions/backup-restore-paseo-state.md.

Worktrees

Worktrees stay pinned under <project>/.paseo-worktrees (a host mount), so their
files already survive recreation; restoring $PASEO_HOME is what makes Paseo
re-recognize them. worktrees.root self-corrects on the next acq run.

What the snapshot excludes

paseo.pid (PID lock), daemon.log (live, churning log), daemon-keypair.json
/ cli-client-id / server-id (per-install identity), and models/
(re-downloadable caches — a speech model alone is ~460 MB).

Live testing found and fixed four bugs

Initial versions passed offline but broke on a real acq (msb) sandbox. Each was
diagnosed live and fixed:

  1. acq exec hangs on raw binary stdin/stdout (guest never sees EOF) →
    transport the tar as base64 text both directions. Backend-agnostic (msb
    and sbx); no reliance on msb-specific --stream.
  2. tar exits 1/2 on live-file churn while still producing a valid archive →
    snapshot in-guest, then classify tar's exit code (only >=3 aborts).
  3. Stale paseo.pid caused a 5s crash-loop after the restore bounce
    (Failed to acquire PID lock; Paseo treats the lock as stale only after 5
    min) → the bounce waits for the daemon to exit, then removes paseo.pid so
    the supervisor relaunch is clean.
  4. 483 MB backup from a models/ cache made base64 crawl → exclude it (and
    server-id).

Also fixed a BSD/macOS mktemp incompatibility (templates must end in the
XXXXXX run) and marked scripts/verify executable.

Verification

RUN_ACQ=1 scripts/verify — all checks passed on a real msb sandbox,
including new step 11 (backup/restore round-trip): valid tar excluding
paseo.pid, sentinel round-tripped into $PASEO_HOME, daemon healed after the
restore-triggered bounce. Offline gate (scripts/verify) and the repo's
unsafe-shell scan (scripts/scan_unsafe_shell.py) also pass.

RUN_ACQ=1 step 11 transcript 
11. Backup/restore round-trip (live, via acq)
  PASS paseo-backup produced a valid tar excluding paseo.pid
  PASS restore round-tripped the sentinel back into \$PASEO_HOME
  PASS daemon healed after the restore-triggered bounce

Summary
  All checks passed.

Rollback

Revert this PR. The scripts are additive host-side tooling; removing them has no
effect on the kit's runtime behavior. Existing backups under
~/.acq-paseo-backups are plain tars and can be deleted at will.

Security impact

No change to auth/authz or attack surface. Scripts are dry-run by default,
read-only against the sandbox on backup, constrain the sandbox-name argument to
a safe charset (SI-10), and transport data only over the existing acq exec
channel to the user's own sandbox.

AI assistance

Authored with AI assistance (OpenCode). The four live-testing fixes were
diagnosed interactively against a real sandbox with the maintainer.

mogul and others added 3 commits September 17, 2026 06:00
@mogul OpenCode Agent
feat(paseo): add backup/restore scripts to preserve state across recr…
502ff79 
…eation

An acq sandbox is ephemeral: only host bind mounts survive teardown, so
recreating one loses $PASEO_HOME (config.json, the projects registry, and
session/agent records) and leaves the on-disk worktrees orphaned in Paseo.

Add two host-side helpers that snapshot Paseo state before teardown and replay
it into the new sandbox afterward, using only acq exec stream I/O (no new acq
capability, no persistent-volume primitive, no coupling of state to a project
repo, no extra mount arg):

- scripts/paseo-backup: streams a tar of $PASEO_HOME (minus paseo.pid) over
  acq exec stdout to a host tar; dry-run by default, atomic temp+rename, validates
  a non-empty tar before promoting.
- scripts/paseo-restore: streams a backup tar back into $PASEO_HOME and bounces
  the daemon (marker-scoped kill) so it re-reads restored config/projects; with
  no tar given it auto-selects the newest backup for the sandbox.

Worktrees stay pinned under the project mount (they already survive recreate);
the restored $PASEO_HOME is what makes Paseo re-recognize them, and worktrees.root
self-corrects on the next acq run.

Includes an ADR (rationale over persistent volumes, in/out of the snapshot, the
worktree analysis, the filename contract), README + TROUBLESHOOTING sections, and
verify assertions (offline dry-run/newest-select/name-guard checks + a live
backup->clear->restore round-trip under RUN_ACQ).

Co-authored-by: OpenCode Agent <agent@gsa.gov>
@mogul OpenCode Agent
fix(paseo): make backup/restore work over real acq exec
fa538d6 
Live testing on an acq (msb) sandbox surfaced four issues in the initial
backup/restore scripts; fix all so the round-trip actually works end-to-end:

- Transport as base64 TEXT, not raw binary. acq exec hangs on a raw binary
  stdin/stdout stream (guest never sees EOF); text reaches EOF normally. The
  guest encodes on backup and the host encodes on restore, keeping the payload
  in acq's safe text channel on both the msb and sbx backends (no reliance on
  msb-specific --stream).
- Snapshot-in-guest then classify tar's exit code. The daemon writes under
  $PASEO_HOME while tar reads it, so GNU tar exits 1/2 ("file changed as we read
  it") while still producing a usable archive; only rc>=3 is a real failure.
- Clear the stale PID lock on the restore daemon bounce. Killing the daemon
  leaves paseo.pid behind, which Paseo only treats as stale after 5 minutes, so
  the 5s supervisor respawn loops forever on "Failed to acquire PID lock". The
  bounce now waits for exit and removes paseo.pid so the relaunch is clean.
- Exclude re-downloadable/identity files so the archive stays small: models/
  (a ~460MB speech-model cache that made base64 crawl), server-id, plus the
  existing daemon.log / paseo.pid / daemon-keypair.json / cli-client-id.
- Also fix a BSD (macOS) mktemp incompatibility: templates must end in the
  XXXXXX run, so the host-side temp names were reordered.

Verified live: backup produces a small valid tar with the right exclusions,
restore round-trips a sentinel back into $PASEO_HOME, and the daemon returns
HEALTH-OK after the bounce. Offline verify gate and unsafe-shell scan pass.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
@mogul OpenCode Agent
fix(paseo): mark scripts/verify executable
a9e4199 
The verify script was tracked as 100644, so a fresh checkout had to chmod +x it
before RUN_ACQ=1 would run. Set the executable bit to match paseo-backup /
paseo-restore.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
@mogul
mogul requested a review from a team as a code owner September 17, 2026 07:04
@mogul OpenCode Agent
fix(paseo): prune logs during state restore
1295191 
Exclude rotated daemon logs from new backups, filter legacy backup logs before restore transport, and swap restored state into place only after a complete extract.

Co-authored-by: OpenCode Agent <agent@gsa.gov>
@mogul
mogul marked this pull request as draft September 21, 2026 17:09
@mogul

mogul commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

Converted back to draft while I consider a lower-level acq subcommand for handling backup/restore in the general case... I'm likely to close this one if I make progress on that one.

@wz-gsa wz-gsa mentioned this pull request Sep 23, 2026
Merged
@mogul

mogul commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

Confirmed: I'm closing this in favor of GSA-TTS/agentic-coding-quickstart#507

@mogul mogul closed this Sep 23, 2026
@mogul
mogul deleted the grey-bulldog branch September 24, 2026 03:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mogul @wz-gsa