Show each machine's CPU, memory and disk load #527

Request: none

Problem

An operator cannot see how loaded a machine is. simlock status, simlock worker list and the console say what a machine is (OS, architecture, tool versions) and what Simlock runs on it (devices, leases, capacity). They do not say how busy its CPU is, how much memory it has left, or how full its disk is.

So when devices boot slowly or an agent's tests time out, the operator has to log in to each machine and open Activity Monitor or top to find the one that is struggling. With a gateway and several workers, that means several logins.

Who it is for

• The operator of a fleet. They watch many workers from one console or one simlock worker list, and want to spot an overloaded machine without logging in to it.
• The owner of a single machine. They run simlock status and want to know whether the machine itself is the reason things are slow.
• Scripts and dashboards. They read --json or the HTTP API and want the same figures.

Outcome

Every place that shows a machine's host facts also shows its current load:

• CPU: how busy it is, as a percentage, over the last 15 seconds, and on how many cores.
• Memory: how much is available out of the total, and how much swap is in use. On macOS, also the memory pressure level the operating system reports.
• Disk: free space out of the total, on the disk that holds Simlock's data folder.
• How old the reading is.

This works on macOS and on Linux, including a Linux container. Where a Linux CPU or memory limit applies to Simlock (a container's limit, or one set on Simlock's own service), that figure is measured against the limit and marked limit. Without a limit, the figures are the whole machine's.

The readings are for people to look at. Nothing in Simlock acts on them.

How it works for the user

flowchart LR
  S[Worker reads its machine every 15 s] --> C[Latest reading]
  C --> ST["simlock status, --json, HTTP"]
  C -->|"gateway's usual refresh"| V[Gateway's view of the worker]
  V --> WL[simlock worker list]
  V --> UI["Console: worker card and detail page"]
  V -->|worker disconnected| X[No reading shown]

simlock status on a macOS worker. Before:

Daemon: running (worker)
Host: macOS 15.5 arm64; xcode 16.4 (16F6)
...

After:

Daemon: running (worker)
Host: macOS 15.5 arm64; xcode 16.4 (16F6)
Load: CPU 34% busy (10 cores) · memory 9.8 GiB of 32.0 GiB available · swap 1.2 GiB used · pressure normal · disk 112.4 GiB of 494.4 GiB free · read 4s ago
...

simlock status in a Linux container limited to 2 CPUs and 8 GiB:

Host: Linux 6.8.0 x64; emulator 35.4.9
Load: CPU 61% busy (2 cores, limit) · memory 3.1 GiB of 8.0 GiB available (limit) · swap 0 B used · disk 40.2 GiB of 120.0 GiB free · read 9s ago

simlock worker list on a gateway. Each connected worker gets its Load line under its Host line. A disconnected worker shows Load: no reading while disconnected.

The console. Each worker card on the Workers page gets a one-line summary (CPU 34% · memory 9.8 of 32.0 GiB · disk 112.4 GiB free). The worker's detail page shows the full readings and their age next to the host facts.

--json and the HTTP API. simlock status --json, GET /v1/status and GET /v1/workers carry the same readings as structured fields.

Examples

What the Load line says in each situation. Today there is no Load line in any of them.

What could go wrong

• The figures differ a little from Activity Monitor or top, which measure over different time windows. Likely, and harmless.
• A Linux machine or container on the older cgroup v1 system shows the whole machine's figures even when a limit is set, with no limit marker. Unlikely on current distributions and Docker.
• If a worker's readings stop updating, the shown age keeps growing, so the operator can tell. A reading never looks fresher than it is.
• Taking a reading every 15 s costs a little CPU on each worker. Expected to be too small to notice.

Words used

• busy %: the share of the CPU time available that was spent working over the last 15 s: the whole machine's, or within the CPU limit when one applies. E.g. CPU 34% busy (10 cores).
• available memory: memory that can be handed out without swapping. It is the figure Activity Monitor and Linux's MemAvailable use, not the smaller "free" figure. E.g. memory 9.8 GiB of 32.0 GiB available.
• memory pressure: macOS's own judgement of how short of memory the machine is: normal, warn or critical. Linux has no equivalent, so it is not shown there.
• limit: a cap on the CPU or memory Simlock and everything it starts may use, set through Linux cgroup v2 (cpu.max, memory.max). A container's limits are one case; a limit on Simlock's own service is another. E.g. 2 cores and 8 GiB. Shown as limit next to the figure it caps.
• Simlock data folder: where Simlock keeps its devices and state, the folder the low-disk cleanup already watches.
• reading age: how long ago the worker took the reading being shown, e.g. read 12s ago.

Non-goals

• Routing on load. The gateway does not pick workers by these readings.
• Acting on load. Capacity, the warm pool and cleanup do not read them.
• New thresholds, warnings or events. No new config key, no "high load" alert, no new event. The existing low-disk cleanup is unchanged.
• A command to start or stop sending readings, or a live stream of them. Readings travel only inside the status answers the gateway already asks for.
• History or charts over time. Each place shows the latest reading only.
• Load per device. How much CPU one simulator or emulator uses is not measured.
• Disks other than the one holding Simlock's data folder, such as where iOS runtimes or Android system images are installed.
• Linux cgroup v1 limits.
• A Prometheus or other metrics export.

Completion conditions

• On a macOS worker, simlock status prints a Load line with CPU busy %, cores, available and total memory, swap used, the pressure level, free and total disk, and the reading's age.
• On a Linux worker that is not in a container, simlock status prints the same Load line without the pressure part.
• Where cgroup v2 CPU and memory limits apply to Simlock, the cores and the memory total equal those limits and both are marked limit. Where none applies, the figures are the whole machine's and carry no marker.
• simlock status --json and GET /v1/status carry the same readings as simlock status prints.
• On a gateway, GET /v1/workers and simlock worker list carry each connected worker's latest readings with their age, and no readings for a disconnected worker.
• The console shows a one-line load summary on each connected worker's card, and the full readings with their age on its detail page.
• A figure that cannot be read shows as unavailable while the others still show, and simlock status still answers.
• simlock status takes no longer with this feature than without it.
• A worker that is still starting shows its Load line.
• A gateway's own simlock status shows the gateway machine's load.
• With the same devices, leases and requests, the gateway picks the same worker and the worker makes the same capacity and cleanup decisions as before this feature.
• No new operation, HTTP route, event or config key is added.
• The readings come from a module of their own, and the daemon behaves the same without it apart from showing no load.

Open questions

Decisions

• ADR 0028 — Host load is an optional module whose readings ride on status: fixes where the code lives, the load wire shape and that readings travel only inside status answers (docs(adr): 0027 host load is an optional module whose readings ride on status #529, Accepted — not yet implemented).

Tasks

flowchart LR
  T1["#531 worker status shows load"] --> T2["#532 Linux cgroup limits"]
  T1 --> T3["#533 gateway carries it"]
  T3 --> T4["#534 console"]

• A worker's simlock status shows its CPU, memory and disk load #531 A worker's simlock status shows its CPU, memory and disk load — a single machine's operator sees its load in simlock status, --json and GET /v1/status, on macOS and Linux. Risk: a wrong or missing Load line, or a daemon that does not exit after a failed start.
• Linux load is measured against a cgroup v2 CPU or memory limit and marked limit #532 Linux load is measured against a cgroup v2 CPU or memory limit — in a container or under a service limit, the figures are the ones Simlock can actually use, marked limit. Risk: a container shows the host's figures, or a limit shows where none is set.
• A gateway carries each worker's load in its view and reports its own #533 A gateway carries each worker's load in its view and reports its own — a fleet operator sees every worker's load in simlock worker list and GET /v1/workers, and none for a disconnected worker. Risk: a stale load on a worker that is gone.
• The console shows each worker's load on its card and detail page #534 The console shows each worker's load on its card and detail page — the operator spots an overloaded machine in the console. Risk: a frozen or missing load on a card.

These tasks run alongside #511's tasks (#519, #513, #514, #515) and #348 and #318 with no ordering, by the maintainer's choice: whichever PR merges second takes the base branch and bumps the protocol again.

Written by an agent.

Generated by Claude Code