feat(core): a quarantined device says what failed

docs/CLI.md

@@ -1413,6 +1413,16 @@ threshold (see `simlock doctor` below): it stays visible in `status` and
 the background (a physical device included, which is unenrolled, never destroyed, when its
 retries run out), and is never handed to a new requester.
 
+A quarantined device says what failed. `status` ends its marker with `;` and the
+cause, for example `(purge retry 0, next retry at 1760619150000; purge failed:
+DriverCrashError: simctl erase exited 164)`. The cause is `purge failed`,
+`shutdown failed` or `delete failed` followed by the tool's error message (cut to
+300 characters), or `stalled while provisioning` or `stalled while reclaiming`,
+which has no message. `status --json` and `list --devices` carry it as
+`quarantineCause` (`kind`, and `stalledState` or `message`). A device quarantined
+before the cause was recorded shows none. The cause goes when the device leaves
+quarantine. A caller with an agent role sees the kind but not the message.
+
 A device currently `provisioning` or `reclaiming` carries a derived
 `transitionAgeMs` — how long it has been in that state — visible in `status`
 and `list --devices` well before it crosses the threshold that would make

docs/CONSOLE.md

@@ -327,7 +327,7 @@ reconnects or is removed. The console only shows these; it does not fix them.
 | `incompatible` | The worker and the gateway have no protocol version in common. Its page says which versions each speaks. |
 | `drained` | The worker gets no new leases (`simlock worker drain`). |
 | `over RAM budget` | The devices on the worker use more RAM than its budget allows, as `simlock status` reports with `(over limit)`. |
-| `quarantined` | A device Simlock could not clean up. It retries the cleanup in the background and leases the device to no one meanwhile. |
+| `quarantined` | A device Simlock could not clean up; its `Cause` says what failed. It retries the cleanup in the background and leases the device to no one meanwhile. |
 | `stalled` | A device stuck `provisioning` or `reclaiming` for longer than it should take, with nothing working on it. `simlock doctor` reports the same devices. |
 
 The first three come from a gateway only: a single host has no gateway to lose
@@ -336,7 +336,9 @@ too.
 
 Select an item to see the facts that matter for it, such as when a
 disconnected worker was last seen or how much RAM a worker uses of its budget,
-and the events of the last hour about that worker or device. **Open worker**
+and the events of the last hour about that worker or device. A quarantined
+device shows its `Cause`: `Purge failed`, `Shutdown failed` or `Delete failed` with the
+tool's error message, or `Stalled while provisioning` or `Stalled while reclaiming`. **Open worker**
 opens the worker's page.
 
 Each item shows within about a second of the daemon reporting it. A stalled

docs/EVENTS.md

@@ -42,7 +42,7 @@ the gateway's own lease index before any worker is ever contacted. `lease-id-tak
 | `device.ready` | device id, boot duration | readiness probe passed (on iOS simulators, boot completion and a successful installer probe) | Registry | implemented |
 | `device.reclaimed` | device id, strategy (erase/snapshot/wipe/uninstall), duration, `missingApps` for a physical device (the apps on its enrollment list that were no longer installed; they are reported, not reinstalled) | fresh-state reclaim finished. Never emitted for a device created under `lease.identity` `fresh`: nothing is reclaimed, the device is deleted instead | Registry | implemented |
 | `device.purge-failed` | device id, lease id, attempted strategy (erase/snapshot/wipe/delete/uninstall), duration, stable error summary | release-time purge failed (including the wipe of a device a daemon start put off; a physical device that reads not present emits nothing and becomes absent), or (strategy `delete`) the shutdown or delete that ends a `fresh` device's lease failed; the device enters `quarantined` (see below) rather than rejoining the pool. The strategy list can grow: a consumer must tolerate a strategy it does not know | Reclaim coordinator | implemented |
-| `device.quarantined` | device id, max retries, next retry deadline | a device committed to `quarantined` — present in the registry, still counted as running when it is a virtual device, not eligible for a grant. Fires immediately after `device.purge-failed` for a release-time purge or delete failure, or on its own for a stalled-transition timeout (see `device.stalled-transition-detected`) or for a returned physical device whose reclaim failed while it was present | QuarantineCoordinator | implemented |
+| `device.quarantined` | device id, max retries, next retry deadline, platform, cause kind (`purge-failed`, `shutdown-failed`, `delete-failed` or `stalled`), stalled state (`stalled` only), message (the other three, at most 300 characters) | a device committed to `quarantined` — present in the registry, still counted as running when it is a virtual device, not eligible for a grant. Fires immediately after `device.purge-failed` for a release-time purge or delete failure, or on its own for a stalled-transition timeout (see `device.stalled-transition-detected`) or for a returned physical device whose reclaim failed while it was present | QuarantineCoordinator | implemented |
 | `device.quarantine-recovered` | device id, attempts, reclaim strategy (erase/snapshot/wipe/uninstall), `missingApps` for a physical device | a quarantined device's retried purge succeeded; it returned to `ready`/`shutdown` and rejoined the warm pool. Never emitted for a `fresh` device: its retry is a delete, and a successful one emits `device.deleted` | QuarantineCoordinator | implemented |
 | `device.quarantine-abandoned` | device id, attempts | a quarantined device exhausted its configured retry budget (`warmPool.quarantine.maxRetries`) — purge retries, or delete retries for a `fresh` device — and was destroyed, or, for a physical device, unenrolled: Simlock never destroys a physical device | QuarantineCoordinator | implemented |
 | `device.quarantine-stranded` | device id, attempts, stable error summary | a quarantined device exhausted its retry budget (purge or delete retries) and the destroy that should have retired it also failed; it stays `quarantined` with no further retry until an operator intervenes | QuarantineCoordinator | implemented |

docs/HTTP-API.md

@@ -288,6 +288,12 @@ past its threshold with nothing working on it also carries
 `stalledTransition.*` in [CONFIGURATION.md](CONFIGURATION.md)). Every other
 device has no `stalled` field, and an older daemon sends none.
 
+A `quarantined` device in `devices` carries **`quarantineCause`**: `kind` (`purge-failed`,
+`shutdown-failed`, `delete-failed` or `stalled`), `stalledState` (`provisioning` or
+`reclaiming`) on `stalled` alone, and `message`, the tool's error text of at most 300
+characters, on the other three. An agent token gets the cause without `message`; an operator
+token gets it whole. A device quarantined before the cause was recorded has none.
+
 Every device in `devices` carries **`servesDefaultMode`**, a boolean: whether
 its pool is the one a lease request with no `mode` draws from on this daemon
 (the default mode of the device's platform). `mode` says what the device is;

docs/internal/EVENTS.md

@@ -80,7 +80,7 @@ worker is ever contacted. `lease-id-taken` (ADR 0020; widens a published vocabul
 | `device.ready` | device id, boot duration | readiness probe passed (on iOS simulators, boot completion and a successful installer probe) | Registry | implemented |
 | `device.reclaimed` | device id, strategy (erase/snapshot/wipe/uninstall, ADR 0022 §4), duration, `missingApps` for a physical device (enrollment-list apps no longer installed: reported, not reinstalled; additive) | fresh-state reclaim finished. Never emitted for a device created under `lease.identity` `fresh` (#75): nothing is reclaimed, the device is deleted instead | Registry | implemented |
 | `device.purge-failed` | device id, lease id, attempted strategy (erase/snapshot/wipe/delete/uninstall), duration, stable error summary | release-time purge failed (including the deferred wipe a daemon start runs in the background, ADR 0019 §2; a physical device read not present emits nothing and becomes `absent`), or (strategy `delete`, #75) the shutdown or delete that ends a `fresh` device's lease failed; the device enters `quarantined` (see below) rather than rejoining the pool. `delete` widens a published vocabulary (events rule 6 allows additive changes): a consumer must tolerate a strategy it does not know | ReclaimCoordinator (strategy `delete`: ReclaimCoordinator's spent-device path) | implemented |
-| `device.quarantined` | device id, max retries, next retry deadline | a device committed to `quarantined` — present in the registry, still counted as running when it is a virtual device, not eligible for a grant. Fires immediately after `device.purge-failed` for a release-time purge or delete failure (`reclaiming`/`shutdown → quarantined`), or on its own for a stalled-transition timeout (see `device.stalled-transition-detected`) or for a returned physical device whose reclaim failed while it was present (ADR 0022 §8: there is no lease for `device.purge-failed` to name) | QuarantineCoordinator | implemented |
+| `device.quarantined` | device id, max retries, next retry deadline, platform, cause kind (`purge-failed`, `shutdown-failed`, `delete-failed` or `stalled`), stalled state (`stalled` only), message (the other three, at most 300 characters) | a device committed to `quarantined` — present in the registry, still counted as running when it is a virtual device, not eligible for a grant. Fires immediately after `device.purge-failed` for a release-time purge or delete failure (`reclaiming`/`shutdown → quarantined`), or on its own for a stalled-transition timeout (see `device.stalled-transition-detected`) or for a returned physical device whose reclaim failed while it was present (ADR 0022 §8: there is no lease for `device.purge-failed` to name) | QuarantineCoordinator | implemented |
 | `device.quarantine-recovered` | device id, attempts, reclaim strategy (erase/snapshot/wipe/uninstall), `missingApps` for a physical device (additive) | a quarantined device's retried purge succeeded; it returned to `ready`/`shutdown` and rejoined the warm pool. Never emitted for a spent `fresh` device (`mayBeGranted` false): its retry is a delete, and a successful one emits `device.deleted` with initiator `lease-end` | QuarantineCoordinator | implemented |
 | `device.quarantine-abandoned` | device id, attempts | a quarantined device exhausted its configured retry budget (`warmPool.quarantine.maxRetries`) — purge retries, or delete retries for a spent `fresh` device — and was destroyed, or, for a physical device, unenrolled: Simlock never destroys one (safety rule 2, ADR 0022 §4) | QuarantineCoordinator | implemented |
 | `device.quarantine-stranded` | device id, attempts, stable error summary | a quarantined device exhausted its retry budget (purge or delete retries) and the destroy that should have retired it also failed; it stays `quarantined` with no further retry until an operator intervenes | QuarantineCoordinator | implemented |

e2e/console/attention.spec.ts

@@ -374,6 +374,9 @@ test.describe("the attention view", () => {
       expect(quarantined?.id).toBeTruthy();
       expect(stalled?.id).toBeTruthy();
 
+      await item(page, "quarantined", `iPhone 16 on ${name}`).click();
+      await expect(facts(page)).toContainText(/Cause\s*Purge failed: Error: erase failed/);
+
       await item(page, "over RAM budget", name).click();
       await expect(facts(page)).toContainText(/Used\s*\d+\.\d\d GiB/);
       await expect(facts(page)).toContainText(/Limit\s*\d+\.\d\d GiB/);

src/bus/index.ts

@@ -155,6 +155,10 @@ export interface EventMap {
     readonly deviceId: string;
     readonly maxRetries: number;
     readonly nextRetryAt: number;
+    readonly platform: "ios" | "android";
+    readonly kind: "purge-failed" | "shutdown-failed" | "delete-failed" | "stalled";
+    readonly stalledState?: "provisioning" | "reclaiming" | undefined;
+    readonly message?: string | undefined;
   };
   "device.absence-detected": {
     readonly deviceId: string;

src/cli/index.test.ts

@@ -4097,6 +4097,62 @@ describe("CLI: status renders the fleet a gateway reports (ADR 0005 §20)", () =
     expect(lines).toContain("Device dev_v1: ready, mode full, serves default mode: yes");
   });
 
+  it("status ends a quarantined device's marker with ; and the cause for each kind, and prints today's marker when there is no cause", async () => {
+    const quarantined = (
+      id: string,
+      quarantineCause?: Record<string, unknown>,
+    ): Record<string, unknown> => ({
+      id,
+      mode: "full",
+      physical: false,
+      quarantineAttempts: 1,
+      quarantineNextRetryAt: 1_760_619_150_000,
+      servesDefaultMode: true,
+      spec: { model: "iPhone 16", osVersion: "18.4", physical: false, platform: "ios" },
+      state: "quarantined",
+      ...(quarantineCause === undefined ? {} : { quarantineCause }),
+    });
+    const status = {
+      ...EMPTY_STATUS,
+      devices: [
+        quarantined("dev_purge", {
+          kind: "purge-failed",
+          message: "DriverCrashError: simctl erase exited 164, then stalled",
+        }),
+        quarantined("dev_shutdown", {
+          kind: "shutdown-failed",
+          message: "Error: shutdown exploded",
+        }),
+        quarantined("dev_delete", { kind: "delete-failed", message: "Error: delete exploded" }),
+        quarantined("dev_prov", { kind: "stalled", stalledState: "provisioning" }),
+        quarantined("dev_recl", { kind: "stalled", stalledState: "reclaiming" }),
+        quarantined("dev_none"),
+      ],
+    } as unknown as StatusGetOutput;
+    const output = outputCapture();
+
+    await runCli(
+      ["status"],
+      output.environmentWith({
+        connectAdmin: async () => fakeClient({ getStatus: () => Promise.resolve(status) }),
+      }),
+    );
+
+    const lines = output.stdout.split("\n");
+    const head =
+      "quarantined, mode full, serves default mode: yes (purge retry 1, next retry at 1760619150000";
+    expect(lines).toContain(
+      `Device dev_purge: ${head}; purge failed: DriverCrashError: simctl erase exited 164, then stalled)`,
+    );
+    expect(lines).toContain(
+      `Device dev_shutdown: ${head}; shutdown failed: Error: shutdown exploded)`,
+    );
+    expect(lines).toContain(`Device dev_delete: ${head}; delete failed: Error: delete exploded)`);
+    expect(lines).toContain(`Device dev_prov: ${head}; stalled while provisioning)`);
+    expect(lines).toContain(`Device dev_recl: ${head}; stalled while reclaiming)`);
+    expect(lines).toContain(`Device dev_none: ${head})`);
+  });
+
   it("status prints absent (<reason>) as the state of an absent device", async () => {
     const physical = (id: string, state: string, absentReason?: string) => ({
       class: "phone",

src/cli/index.ts

@@ -438,6 +438,7 @@ function extractGlobalToken(argv: readonly string[]): {
   return { token, rest };
 }
 
+// fallow-ignore-next-line complexity -- one dispatcher over every command, unchanged by this task; the file is touched for the status marker.
 export async function runCli(
   argv: readonly string[],
   environment: CliEnvironment = defaultCliEnvironment(),
@@ -2169,7 +2170,18 @@ function quarantineMarker(device: NonNullable<StatusGetOutput["devices"]>[number
     device.quarantineNextRetryAt === undefined
       ? ""
       : `, next retry at ${device.quarantineNextRetryAt}`;
-  return `purge retry ${attempts}${nextRetryAt}`;
+  const cause = device.quarantineCause;
+  return `purge retry ${attempts}${nextRetryAt}${cause === undefined ? "" : `; ${quarantineCauseText(cause)}`}`;
+}
+
+function quarantineCauseText(cause: {
+  readonly kind: "purge-failed" | "shutdown-failed" | "delete-failed" | "stalled";
+  readonly stalledState?: string | undefined;
+  readonly message?: string | undefined;
+}): string {
+  if (cause.kind === "stalled") return `stalled while ${cause.stalledState ?? ""}`;
+  const text = cause.kind.replace("-", " ");
+  return cause.message === undefined ? text : `${text}: ${cause.message}`;
 }
 
 const DEVICE_CLASS_ORDER: readonly string[] = deviceClassSchema.options;

src/contract/AGENTS.md

@@ -55,7 +55,7 @@ Three identities:
 
 ## Protocol version
 
-- Now `{ min: 39, max: 39 }`. `DAEMON_PROTOCOL_VERSION` (`src/daemon-protocol/`) is its `max`.
+- Now `{ min: 40, max: 40 }`. `DAEMON_PROTOCOL_VERSION` (`src/daemon-protocol/`) is its `max`.
 - Every change to a contract shape bumps it by one in the same PR. Move `min` too, unless
   a compatibility path stays side by side.
 - `hello` picks the highest overlap. None → `PROTOCOL_VERSION_UNSUPPORTED` with both ranges.

src/contract/protocol.test.ts

@@ -47,7 +47,7 @@ describe("helloRequestSchema", () => {
 });
 
 describe("PROTOCOL_VERSION_RANGE", () => {
-  it("is one above the base branch's, for agent-device's section moving into config.get's capabilities", () => {
-    expect(PROTOCOL_VERSION_RANGE).toEqual({ max: 39, min: 39 });
+  it("is one above the base branch's, for a quarantined device's cause on status.get and list.get", () => {
+    expect(PROTOCOL_VERSION_RANGE).toEqual({ max: 40, min: 40 });
   });
 });

src/contract/protocol.ts

@@ -10,7 +10,7 @@ export interface ProtocolRange {
 const protocolRangeSchema = z.object({ min: z.number().int(), max: z.number().int() });
 
 /** Widen only once a second version is kept alive side by side with the first; a bump that leaves no shim behind moves both ends. */
-export const PROTOCOL_VERSION_RANGE: ProtocolRange = { min: 39, max: 39 };
+export const PROTOCOL_VERSION_RANGE: ProtocolRange = { min: 40, max: 40 };
 
 const LEGACY_DAEMON_PROTOCOL_VERSION = 2;
 

src/contract/schemas.test.ts

@@ -44,6 +44,7 @@ describe("leaseGrantSchema's device projection", () => {
     "quarantinedAt",
     "quarantineAttempts",
     "quarantineNextRetryAt",
+    "quarantineCause",
     "leaseIdentity",
     "transitionAgeMs",
   ] as const;
@@ -67,6 +68,7 @@ describe("leaseGrantSchema's device projection", () => {
       quarantinedAt: 50,
       quarantineAttempts: 3,
       quarantineNextRetryAt: 60,
+      quarantineCause: { kind: "purge-failed", message: "Error: boom" },
       address: "127.0.0.1:1234",
       mode: "slim",
       leaseIdentity: "fresh",
@@ -261,6 +263,43 @@ describe("statusDeviceSchema's device projection", () => {
     }
   });
 
+  it("statusDeviceSchema and deviceRecordSchema accept each of the four quarantine cause kinds and refuse every malformed cause", () => {
+    const device = { ...fullCoreShapedDevice(), servesDefaultMode: true };
+    const accepted = [
+      { kind: "purge-failed", message: "DriverCrashError: simctl erase exited 164" },
+      { kind: "shutdown-failed", message: "Error: shutdown exploded" },
+      { kind: "delete-failed", message: "x".repeat(300) },
+      { kind: "purge-failed" },
+      { kind: "stalled", stalledState: "provisioning" },
+      { kind: "stalled", stalledState: "reclaiming" },
+    ];
+    const refused = [
+      { kind: "exploded", message: "Error: x" },
+      { kind: "stalled" },
+      { kind: "purge-failed", message: "Error: x", stalledState: "reclaiming" },
+      { kind: "stalled", message: "Error: x", stalledState: "reclaiming" },
+      { kind: "stalled", stalledState: "ready" },
+      { kind: "purge-failed", message: 7 },
+      { kind: "purge-failed", message: "" },
+      { kind: "purge-failed", message: "x".repeat(301) },
+      { kind: "purge-failed", message: "Error: a\nb" },
+      { kind: "purge-failed", message: "Error: a\u2028b" },
+    ];
+
+    for (const schema of [statusDeviceSchema, deviceRecordSchema]) {
+      for (const quarantineCause of accepted) {
+        expect(schema.parse({ ...device, quarantineCause })).toMatchObject({ quarantineCause });
+      }
+      expect(schema.parse(device)).not.toHaveProperty("quarantineCause");
+      for (const quarantineCause of refused) {
+        expect(
+          () => schema.parse({ ...device, quarantineCause }),
+          JSON.stringify(quarantineCause),
+        ).toThrow();
+      }
+    }
+  });
+
   it("rejects a device object that is missing the fields status.get must keep", () => {
     expect(() => statusDeviceSchema.parse({ id: "device-1" })).toThrow();
   });