feat(sandboxd): add runtime catalog and live volume support

.github/workflows/sandboxd.yml

@@ -67,6 +67,29 @@ jobs:
       - name: Test
         run: make go-test
 
+      - name: Test pool as root
+        working-directory: sandboxd
+        shell: bash
+        run: |
+          GOWORK=off go test -race -c -o "$RUNNER_TEMP/catalog.test" ./pool
+          sudo setpriv --bounding-set=-dac_override,-dac_read_search "$RUNNER_TEMP/catalog.test" -test.count=1 -test.v -test.skip '^Test(PromoteFailsClosedOnMetaError|NodesOfACellShareThePoolSet|AMetaStoreNodeWithoutACellKeepsItsOwnPools|TwoNodesOnOneMetaStoreShareTheTenantSet)$' | tee "$RUNNER_TEMP/catalog-test.log"
+          grep -q '^--- PASS: TestRuntimeCatalogPersistsNormalizedSemantics ' "$RUNNER_TEMP/catalog-test.log"
+          if grep -- '--- SKIP:' "$RUNNER_TEMP/catalog-test.log"; then
+            exit 1
+          fi
+
+      - name: Test e2e as root
+        working-directory: e2e
+        shell: bash
+        run: |
+          GOWORK=off go test -race -c -o "$RUNNER_TEMP/volume-e2e.test" .
+          sudo setpriv --bounding-set=-dac_override,-dac_read_search "$RUNNER_TEMP/volume-e2e.test" -test.count=1 -test.v -test.skip '^TestCaddyTLSCluster$' | tee "$RUNNER_TEMP/volume-e2e-test.log"
+          grep -q '^--- PASS: TestRuntimeVolumeControllerAdmissionThroughHTTP ' "$RUNNER_TEMP/volume-e2e-test.log"
+          grep -q '^--- PASS: TestRuntimeVolumeExplicitACLAndStaticPublicThroughHTTP ' "$RUNNER_TEMP/volume-e2e-test.log"
+          if grep -- '--- SKIP:' "$RUNNER_TEMP/volume-e2e-test.log"; then
+            exit 1
+          fi
+
       - name: Test the PostgreSQL meta store
         working-directory: sandboxd
         shell: bash
@@ -75,7 +98,9 @@ jobs:
         run: |
           GOWORK=off go test -race -count=1 -v ./tenants/... ./poolset/... | tee "$RUNNER_TEMP/pg-test.log"
           GOWORK=off go test -race -count=1 -v -run 'TestTwoNodesOnOneMetaStore|TestNodesOfACell|TestAMetaStoreNodeWithoutACell' ./pool/ | tee -a "$RUNNER_TEMP/pg-test.log"
-          ! grep -- '--- SKIP' "$RUNNER_TEMP/pg-test.log"
+          if grep -- '--- SKIP:' "$RUNNER_TEMP/pg-test.log"; then
+            exit 1
+          fi
 
       - name: Set up Python
         uses: actions/setup-python@v7.0.0

README.md

@@ -35,7 +35,7 @@ performance) — source in
 - `sandboxd/` — per-node control plane (Go): warm pools refilled from golden
   snapshot exports (online-retunable), claim/release/renew/hibernate/fork/
   promote/checkpoint HTTP API, operator-catalog dataset volumes (read-only or
-  writable; read-only ones change on a running claim), signed preview URLs,
+  writable, mounted or attach-only; all support live changes), signed preview URLs,
   the HTTP-upgrade byte relays to silkd and to any guest port, read-only file
   access to hibernated or archived claims from a warm VM of their pool
   (opt-in per pool `reader`), claim env with

docs/deploy.md

@@ -167,6 +167,34 @@ fragment the warm pools):
 | `xlarge` | 4 | 8G |
 | `2xlarge` | 8 | 16G |
 
+### Disabling archive failover
+
+Before changing `archive_failover`, verify local claim ownership and the
+corresponding remote rows from a node with registry access.
+Set `replicate_seconds: 0` while `archive_failover` is still enabled, then
+restart, so the replica scan retires existing replicas. Disabling
+`archive_failover` does not consume remote rows or fence other nodes. Drain
+this node's rows from a node that still has the registry, or keep peers that
+run with `archive_failover` from adopting its claims.
+
+Disable `archive_failover` only after those replicas have retired. If claims
+still carry `ReplicaCk`, `ReplicaPending`, or `ArchivePending`, startup without
+a registry retains those fields and refuses new live writable volume additions
+with 409 while any of them remain. Release and default destroy-on-expiry leave
+replica checkpoints in the checkpoint store. A successful local archive,
+including archive-on-expiry, clears `ReplicaCk` and `ArchivePending` and queues
+the published replica checkpoint for deletion. It retains `ReplicaPending`,
+which continues to refuse live writable additions after wake. Once the current
+fields clear, recovery checkpoint references can still require temporary 503
+retirement before admission.
+
+Re-enabling `archive_failover` does not guarantee recovery of a retained claim.
+Before re-enabling it, verify that a published replica's row still exists,
+belongs to this node, and can be retired. If the row is absent, was drained, or
+has moved to another owner, choose an explicit recovery procedure before the
+transition. Startup fencing can drop the local claim, and reconciliation can
+remove its VM, losing changes since the last checkpoint.
+
 ### Reading sleeping claims without waking them
 
 A client that only lists or reads files (a UI showing an idle agent's
@@ -247,7 +275,14 @@ the pool had `reader`, and its reads wake the claim as without the feature.
 
 ### Dataset volumes
 
-Running claims can change mounted read-only catalog entries through the
+Volume `tenants` entries in static configuration must match the tenant name
+syntax used by the tenant API. Configurations containing invalid names that
+previously only produced warnings now fail validation at startup. Remove or
+correct those entries before upgrading. Valid names for tenants that have not
+yet been created remain accepted.
+
+Running claims can add or remove mounted and attach-only catalog entries in
+read-only or writable mode through the
 [GET, PUT, and PATCH volume API](sandboxd-api.md#get-v1sandboxesidvolumes)
 and both SDKs. The image must already exist on the owner node. Live changes
 require Cocoon v0.7.2 or newer for idempotent disk attach and detach. PUT and
@@ -260,8 +295,8 @@ The claim journal stores confirmed applied `Volumes`, plus desired
 `VolumesWant` while it differs.
 Admission holds cover their union across daemon restart and teardown. Retry
 an interrupted operation with the same PUT target. Release cancels convergence
-and transfers holds to teardown. Capture is refused while either set is
-nonempty.
+and transfers holds to teardown. Capture is refused while either set, an active
+operation, or unacknowledged recovery references still hold volumes.
 
 A sandbox whose VMM exits while holding volumes is release-only. Release it and
 create a new sandbox. Daemon restart recovery assumes the original VMM remains
@@ -272,8 +307,9 @@ interrupted diagnostic after restart and require a caller retry.
 exercises GET, repeated full-set PUT, PATCH add and remove, and clearing a new
 claim.
 
-Each catalog path must name a disk image containing a mountable whole-device
-filesystem. Keep catalog names bound to the same image while any claim holds
+Mounted entries need a disk image containing a mountable whole-device
+filesystem. Attach-only entries leave device use and filesystem consistency
+to the caller. Keep catalog names bound to the same image while any claim holds
 them. Use versioned names such as `dataset-v1` and
 `dataset-v2`; sandboxd cannot detect a repointed catalog entry. A read-only
 entry's image must stay immutable: do not replace,
@@ -304,18 +340,20 @@ A volume claim may consume an ordinary warm Cloud Hypervisor VM. sandboxd
 attaches after the warm pop or provision, waits for the device settle — a
 serial match under `/sys/block`, then the `/dev/<name>` node itself, since the
 kernel publishes sysfs before devtmpfs creates it — up to 2 seconds, then
-mounts the device before finalizing the claim: `mode: "ro"` (the default)
-attaches and mounts read-only; `mode: "rw"` requires the catalog entry's
-`writable: true` and attaches and mounts read-write. Setup failure destroys
+mounts each managed device before finalizing the claim. `mode: "ro"` (the default)
+attaches read-only; `mode: "rw"` requires the catalog entry's
+`writable: true` and attaches read-write. Per-entry `attach_only: true`, or
+claim-wide `volumes_attach_only: true`, skips the device wait and mount. The
+caller must wait for enumeration before using that device. Setup failure destroys
 the VM without quiescing — the claim was never handed out, so no workload
 write happened — and a popped warm VM is refilled normally.
 
 A multi-volume claim brings every volume up concurrently: each volume's own
 marker→attach→mount order is preserved, but cocoon serializes the hypervisor
 attach per VM, so what actually overlaps across volumes is the CLI spawns,
 device settles, and guest mounts. A partial failure still fails the whole
-claim and destroys the VM without quiescing, so any `rw` volume already
-attached keeps its dirty marker — including one the caller never got a handle
+claim and destroys the VM without quiescing, so any mounted `rw` volume already
+attached keeps its dirty marker, including one the caller never got a handle
 to; recovery is the normal `rw` cycle (see
 [Writable dataset volumes](#writable-dataset-volumes)).
 
@@ -331,19 +369,20 @@ boot.
 the answering node's current local availability and whether the entry is
 `writable`, without exposing host paths or node addresses. Applied names,
 effective mounts, and (for `rw` entries) mode are persisted with the claim.
-Such a claim cannot hibernate, fork, checkpoint, or promote; the idle
-hibernate sweep leaves it running. Release removes the VM but never deletes
-the operator-owned backing image — an `rw` release quiesces (unmounts) first,
+While volume holds or unacknowledged recovery references remain, a claim cannot
+hibernate, fork, checkpoint, or promote. The idle hibernate sweep leaves it
+running. Release removes the VM but never deletes
+the operator-owned backing image. A mounted `rw` release quiesces (unmounts) first,
 below. With `directio=off`, readers share the host page cache; use
 `directio=on` when cache interference matters.
 A dataset mounted into an egress-lane sandbox can be uploaded wherever that
 tenant's egress policy permits, so treat the ACL and egress policy as one access
-decision. Image replication, detach, and refcounting remain out of scope.
+decision. Image replication and business ownership remain external to sandboxd.
 
 #### Writable dataset volumes
 
-A catalog entry with `writable: true` may be claimed with `mode: "rw"`
-(omitted, or `"ro"`, always mounts read-only, even against a writable entry).
+A catalog entry with `writable: true` may be claimed with `mode: "rw"`.
+Omitted mode, or `"ro"`, attaches read-only, even against a writable entry.
 `directio` still defaults to `off` for a writable entry — the durability
 nuance is that the host page cache sits between the guest's flush and the
 device, so `directio=on` is the knob when that gap matters, not just cache
@@ -356,7 +395,7 @@ so every claim for that name, `ro` or `rw`, already funnels to that one node;
 running the same writable name from two nodes at once is dataset divergence
 that nothing in the protocol detects for you.
 
-**The `<path>.dirty` marker.** Before the first `rw` attach of an image,
+**The `<path>.dirty` marker.** Before a mounted `rw` attach of an image,
 sandboxd durably creates a `<path>.dirty` sidecar beside it; a clean `rw`
 release removes it. It is a write-ahead record, not a lock: it survives a
 killed sandboxd or a dead VMM, so anything short of a clean unmount leaves it
@@ -369,6 +408,13 @@ can serve `ro` again. A `ro` claim against a dirty image is refused rather
 than auto-healed: replaying the journal takes the write lock, which conflicts
 with concurrent readers.
 
+Live removal clears a mounted writer's marker only with current-call evidence
+of the expected device at the managed root mount, writable mount and
+superblock options, normal unmount, no remaining device mounts, and confirmed
+host detach. Marker cleanup precedes release of the old hold. A prior external
+unmount or absent device provides no such evidence. Detach can succeed while
+the marker remains and read-only use still requires recovery.
+
 **Concurrency, for one image name:**
 
 - `rw` ∥ `rw` — refused (409).
@@ -380,17 +426,13 @@ with concurrent readers.
   proceed; a crashed writer leaves it, and `ro` claims are refused until one
   `rw` claim replays and releases cleanly.
 
-**Attach-only claims opt out of the marker, not out of admission.** A claim
-sent with `volumes_attach_only` gets the device attached and nothing else, so
-sandboxd neither writes nor clears `<path>.dirty` for it — it cannot verify a
-mount it did not perform. Everything in this section still applies to the
-default, mounting claims, and the exclusion rules above apply to attach-only
-claims exactly the same way, which is what keeps other tenants safe. The
-operator-visible difference: an image whose attach-only writer released
-without unmounting cleanly carries no marker, so the next `ro` claim is
-admitted and fails at mount time (500) instead of being refused early (409).
-Whoever hands out attach-only `rw` access owns that trade; see
-[sandboxd-api](sandboxd-api.md#attach-only-volumes).
+**Attach-only entries have no daemon-managed dirty marker.** This applies to
+both initial claims and live additions. sandboxd neither writes nor clears
+`<path>.dirty` for attach-only `rw`, including a guest crash during writes.
+Reader and writer exclusion still applies, but a later `ro` claim can be
+admitted without a marker and may see an inconsistent filesystem. Mount failure
+is not guaranteed. The controller owns quiescence, unmount, crash recovery,
+and safe read-only publication. See the [attach-only contract](sandboxd-api.md#attach-only-volumes).
 
 ### Reclaiming S3 storage
 

docs/live-volumes.md

@@ -7,7 +7,22 @@ Register both read-only catalog entries on the owning node and grant the
 sandbox's tenant access to them. Put `v1` and `v2` in each image's `/version`
 file. Keep each name bound to the same immutable image while held. Use Cocoon
 v0.7.2 or newer and set `SANDBOX_ADDR` and `SANDBOX_API_TOKEN` for the client.
-A sandbox token alone cannot manage volumes.
+A sandbox token alone cannot manage volumes. For a complete sleep and wake sequence,
+see [the external controller example](volume-controller.md).
+
+A configured node root token can [register backing paths](sandboxd-api.md#put-v1volumesname)
+without editing static config. Registration persists only node-local catalog
+metadata and does not manage disk files. An empty runtime `tenants` list allows
+only root to initiate attachment, including attachment to a tenant-owned claim.
+Keep `api_token` configured. List tenants explicitly to grant self-service;
+that permission also allows reattachment to another claim after detach. For a sleep and wake workflow, an
+external controller owns quiescence and unmount before detach, a reservation
+through sleep, and access gating until wake, attach, and mount complete. That
+gate covers API and preview access on every node that can wake the sandbox.
+Wake establishes only silkd readiness. The [controller example](volume-controller.md)
+has SDK and HTTP contract tests. The controller still supplies the reservation,
+guest operations, and access gate. Live changes support mounted and attach-only
+entries in read-only or writable mode.
 
 Pause application reads before replacement. Close files, mmap mappings, cwd,
 and bind mount references on the old disk. Replacement unmounts the old disk
@@ -58,9 +73,9 @@ with client.new("rt:24.04", ttl_seconds=900) as sb:
 
 A successful run reads `v1`, then `v2`, and preserves every identity assertion.
 The repeated full target verifies that a settled mount can be retained.
-This example starts without other volumes. If your sandbox has writable or
-attach-only volumes, retain them in a full PUT or use PATCH to replace only
-the dataset names. `None`, a nil Go slice, and a missing `volumes` field do not
+This example starts without other volumes. A full PUT replaces the entire
+desired set, including writable and attach-only entries. Retain other entries
+in that list, or use PATCH to replace only the dataset names. `None`, a nil Go slice, and a missing `volumes` field do not
 mean an empty target.
 
 Go uses the same owning handle and client API token.
@@ -91,3 +106,51 @@ admission holds. Detailed errors survive only within one daemon lifetime.
 If the VMM exits, a sandbox holding volumes is release-only. Release it and
 create a new sandbox; neither PUT with an empty target nor daemon restart
 restores the original processes.
+
+## Writable and attach-only entries
+
+Use `mode: "rw"` for a catalog entry that permits writing. Set per-entry
+`attach_only: true` when the caller manages the device. PUT and PATCH accept
+all four modes in a mixed list:
+
+```json
+{"volumes":[{"name":"dataset"},{"name":"scratch","mode":"rw"},{"name":"raw-data","attach_only":true},{"name":"raw-work","mode":"rw","attach_only":true}]}
+```
+
+Omitting a mount still selects `/volumes/<name>` for a new mounted entry. An
+attach-only entry cannot supply a nonempty mount. HTTP responses keep the
+legacy `{name, mount?, mode?}` fields and omit `attach_only`. The current SDKs
+infer the flag from a mountless response. Echoing that response to the same
+sandbox retains the held device-only entry. An omitted or false flag does not
+change that held shape. A mount or mode change requires a successful removal
+before adding the name again.
+
+The existing Go claim option `WithVolumesAttachOnly()` and Python `mount=False`
+remain claim-wide aliases. The current source SDKs also encode an all-attach-only
+initial list with the legacy global flag. They reject mixed mounted and
+attach-only initial claims before network I/O. Claim without volumes, as in the
+example above, then use the live API for a mixed list. Direct HTTP mixed initial
+claims require matching daemons across the serving fleet and redirect targets.
+Older claim decoders can silently ignore per-entry flags. See
+[version compatibility](sandboxd-api.md#volume-version-compatibility). These
+per-entry SDK features are unreleased. Published Python 0.1.19 can still echo a
+same-sandbox GET response but cannot construct the new mixed request mappings.
+
+Before removal, stop application access and close references. For attach-only
+devices, unmount the caller-managed filesystem first. sandboxd checks its guest
+mount namespace, but cannot prove that raw block users, partitions, or other
+mount namespaces are idle. Keep the control-plane reservation across any sleep
+and gate access until wake, attach, and mount complete.
+
+For mounted writable entries, sandboxd clears its dirty marker only after
+identifying the expected device at the managed root mount with writable mount
+and superblock options, completing a normal unmount, confirming no residual
+device mounts, and confirming host detach. The same removal must supply all
+that evidence before the old hold is released.
+External pre-unmount, an absent device, or a lost result leaves the
+marker conservatively in place. Detach can succeed while a subsequent read-only
+attach still requires recovery. Attach-only writers have no daemon-managed
+marker, including after a guest crash during writes. The caller owns recovery
+and must establish consistency before read-only publication. See the [API
+contract](sandboxd-api.md#patch-v1sandboxesidvolumes) for failure and retry
+semantics.