A worker's simlock status shows its CPU, memory and disk load #531

Part of #527.

Scope

A worker reads its own machine every 15 s, and simlock status, simlock status --json and GET /v1/status show that load. This covers macOS, and Linux without limits. Linux limits are #532, so a Linux reading here is always the whole machine's.

This task builds the whole path once: the port with its two adapters, the src/host-load/ module, the status.get field and the CLI line. It has more Tests lines than the usual 30. It stays whole because no part of it can merge alone: a port or a sampler that nothing calls is dead code, which Fallow rejects.

In short

simlock status gains a Load: line under Host:. The daemon reads CPU, memory and disk every 15 s and answers from the latest reading, so status never waits for one.

flowchart LR
  P["HostLoadProbe (new)"] --> S["host-load sampler (new)"]
  S -->|"load()"| D["status.get (changed)"]
  D --> C["simlock status (changed)"]

Before:

Daemon: running (worker)
Host: macOS 15.5 arm64; xcode 16.4 (16F6)

After:

Daemon: running (worker)
Host: macOS 15.5 arm64; xcode 16.4 (16F6)
Load: CPU 34% busy (10 cores) · memory 9.8 GiB of 32.0 GiB available · swap 1.2 GiB used · pressure normal · disk 112.4 GiB of 494.4 GiB free · read 4s ago

If this goes wrong, a user sees a missing or wrong Load: line, a slow simlock status, or a daemon that does not exit.

Technical spec

Follow ADR 0028 §1–§4 (PR #529). This task makes these choices inside it.

Modules touched

flowchart LR
  main["daemon/main.ts *"] --> HL["host-load/ * (new)"]
  main --> F["ports/host-load.ts * (new): productionHostLoadProbe"]
  main --> SRV["daemon/server.ts *"]
  SRV --> WD["daemon/dispatcher.ts *"]
  HL --> F
  F --> FS["ports/filesystem.ts *"]
  F --> PR["ports/process-runner.ts"]
  main --> K["contract: schemas.ts (fitHostLoad), operations.ts, protocol.ts *"]
  CLI["cli/index.ts *"] --> K

• src/ports/host-load.ts. It holds HostLoadProbe, HostLoadCounters, DarwinHostLoadProbe, LinuxHostLoadProbe, FakeHostLoadProbe and productionHostLoadProbe({ platform, dataDirectory }), all exported from src/ports/index.ts.
  • The factory returns the macOS adapter for darwin and the Linux one otherwise. It gives each adapter its production inputs: a NodeProcessRunner of its own (never the daemon's logging runner, so a sysctl every 15 s writes nothing to the daemon log), a NodeFilesystem, os.cpus, and os.cpus().length for Linux cores. os.availableParallelism() follows CPU affinity, while /proc/stat covers every CPU.
  • CPU counters are cumulative busy and total time, in one unit per source. Only their ratio between two reads matters.
    • Linux /proc/stat cpu line: busy is user + nice + system + irq + softirq + steal. Total is busy + idle + iowait. guest and guest_nice are already inside user and nice.
    • macOS os.cpus() times: busy is user + nice + sys + irq. Total is busy + idle.
  • macOS memory. The adapter runs sysctl -n hw.memsize kern.memorystatus_level kern.memorystatus_vm_pressure_level vm.swapusage, with a 5 s process timeout. The output is one value per line, in that order. Example:

    34359738368
    31
    1
    total = 2048.00M  used = 1228.75M  free = 819.25M  (encrypted)
    

    • Available is hw.memsize × level ÷ 100, where level is an integer 0–100.
    • Swap used is the used = <number><K|M|G> field, in binary units (M = 1024²).
    • Pressure is 1 → normal, 2 → warn, 4 → critical. Any other pressure line, numeric or not, leaves only pressure out.
    • A non-zero exit, a timeout, or an unparsable hw.memsize, level or swap line leaves the whole memory part out.
  • Linux memory. MemTotal, MemAvailable, SwapTotal and SwapFree (kB) from /proc/meminfo. limited: false on CPU and memory.
  • Parts. Each part (cpu, memory, disk) is read on its own. A part that fails is left out, and read() never rejects.
• src/ports/filesystem.ts. diskSpace(path): Promise<{ freeBytes, totalBytes }> sits beside diskFree. NodeFilesystem computes both from one statfs: bavail × bsize and blocks × bsize. MemoryFilesystem answers its configured free bytes and a total no smaller than them. Every other Filesystem implementation gains it too: the object literals in src/daemon/admin-secret.test.ts (about :44, :128, :212).
• src/host-load/ (new, entered through index.ts). HostLoadSampler, with start(), current() and dispose():
  • start() reads at once, then on a Clock timer every 15 s, re-armed each time it fires.
  • One read at a time. A tick is skipped while a read is in flight. Each read is bounded at 10 s on the clock, like HostFactsReader's bound. After 10 s the read counts as failed for the reading: every part is absent, and its late answer is ignored. It still counts as in flight until it settles, so a read that hangs for ever holds one call, not a new one every 15 s.
  • Busy %. 100 × Δbusy ÷ Δtotal against the previous successful CPU counters, clamped to 0–100. CPU is absent when there are no previous counters, when Δtotal ≤ 0, and when cores or limited changed since the last read.
  • A probe that rejects or throws anyway (the adapters promise not to) counts as a failed read: every part is absent, the read is no longer in flight, and the sampler logs one warn with the error.
  • Logging. A part is "read" when the probe returned it, and "failed" when it did not. For CPU that means the probe returned no counters. A CPU reading with no busy % (first read, Δtotal ≤ 0, a change of cores or limited) is not a failure. Each part starts as "never read", which logs nothing. A part that was read and then fails logs one warn, naming the part. One that is read again after failing logs one warn too.
  • current() returns { ageMs, cpu?, memory?, disk? }. ageMs is now − the time of the read, clamped at 0 because the wall clock can step back. Before the first read finishes it returns undefined.
  • dispose() cancels the timer. A read that finishes after dispose() changes nothing and logs nothing.
• src/contract/schemas.ts. hostLoadSchema (ADR 0028 §4's bounds; ageMs is an integer 0–86 400 000) and fitHostLoad, beside fitHostFacts. fitHostLoad caps ageMs at one day and drops any part whose figures break the schema, so the daemon's own output check never fails on a reading.
• Lint (.oxlintrc.json).
  • A src/host-load/** override bans fs, fs/**, node:fs, node:fs/**, child_process, child_process/**, node:child_process and node:child_process/**. It also bans imports from core, leasing, gateway, drivers, contract, http, cli, mcp and daemon.
  • The override repeats the top-level index-only patterns, as every existing override does, because an override replaces them.
  • The src/core/**, src/leasing/**, src/drivers/** and src/gateway/** overrides each ban importing any file of host-load, its index.js included. That is the way core, leasing and drivers ban agent-device today; the gateway's agent-device pattern bans only non-index files, so the gateway's host-load pattern must not be copied from it.
  • The src/contract/** override already bans every .. import, which covers host-load.
  • src/host-load/boundary.test.ts is modelled on src/core/warm-pool/boundary.test.ts.
• src/daemon/server.ts.
  • DaemonServerCommonOptions, the options both modes pass, gains hostLoad?: { start(): void; dispose(): void }. A gateway carries each worker's load in its view and reports its own #533's gateway passes it there, with no change to this file.
  • DaemonServerEngineOptions, the worker-only options, gains load?: () => StatusLoad | undefined. buildDispatcher passes load to the dispatcher, beside hostFacts.
  • start() calls hostLoad.start() right after the host has bound its socket, before convergence.
  • hostLoad.dispose() runs in #stop, and whenever start() throws after hostLoad.start() ran: for example, adminSecret.persist() rejecting after the socket is bound. startDaemon does not call stop() on a rejected start, and a pending timer would keep the process alive.
• src/daemon/main.ts. StartDaemonOptions gains hostLoadProbe?: HostLoadProbe | null.
  • Omitted means productionHostLoadProbe({ platform: process.platform, dataDirectory }). null means no sampler: no hostLoad and no load reach DaemonServer.
  • Worker mode builds the sampler after the host facts reader. It passes hostLoad: sampler and load: () => fitHostLoad(sampler.current()). Gateway mode is A gateway carries each worker's load in its view and reports its own #533.
  • Existing tests.
    • The start() helper in src/daemon/main.test.ts passes hostLoadProbe: null unless a test sets one.
    • Tests that call startDaemon directly keep the production probe, which touches neither their ScriptedProcessRunner (it has its own runner) nor anything they assert. The exceptions are the two that count pending timers on their FakeClock (about :747 and :1015): they pass hostLoadProbe: null.
    • e2e/status-while-starting.test.ts asserts a starting worker's status keys are exactly daemon and host. It changes to accept load beside them, and to check load's shape when present.
• src/daemon/dispatcher.ts. #statusGet puts load on the starting answer and the running one when load() returns a value.
• src/cli/index.ts. formatStatus prints the Load: line right after Host:, on the starting path and the full path. The formats, matching the inputs table:
  • Bytes: 0 B for 0, whole MiB below 1 GiB, and GiB with one decimal from 1 GiB up, each rounded half up. A value that rounds to 1024 MiB prints as GiB.
  • Busy %: a whole number, rounded half up.
  • Cores: up to one decimal, or two below 0.1 (a tiny cgroup quota), and <0.01 below 0.005, with core singular at exactly 1.
  • Age: whole seconds rounded up, as read Ns ago.
  • limit goes inside the cores parentheses for CPU, and in its own (limit) after memory.
  • The CLI's own formatGibibytes (two decimals, the RAM budget line) is not changed.
  • The formatter is one unexported function in src/cli/index.ts, which A gateway carries each worker's load in its view and reports its own #533's formatWorkers in the same file calls too.

Contract and event changes

• status.get output gains load: hostLoadSchema.optional(). This PR bumps the protocol by one. That means src/contract/protocol.ts and the pins in src/daemon-protocol/index.test.ts, src/daemon/server.test.ts and src/gateway/service.test.ts.
• No event changes. Searched: no event payload carries host facts.
• feat(fleet): workers advertise their capabilities in status and worker list #519 (PR feat(fleet): workers advertise their capabilities in status and worker list #520) also changes #statusGet, the status.get schema and the protocol. feat(fleet): a lease request may require a capability #513, GET /v1/stats and the simlock stats --csv ledger #348 and Every device shape says when the device entered its state #318 also bump the protocol. All run at the same time by the maintainer's choice: whichever merges second takes the base branch and bumps again.

Other code on the same state

• CleanupReaper reads diskFree on the same data folder on its own 60 s tick, and DiskSpaceGuard reads it before a download. All three only read, so there is no conflict and nothing wins. Disk pressure stays the reaper's own.
• HostFactsReader fills host in the same answer. The two fields are independent, and both are served from memory.
• The capacity strategy's RAM budget (NodeSystemStats.totalRamBytes() minus 4 GiB) is in the same answer under capacity.ramBudget. The Load memory total may differ from it. Neither changes the other.
• The daemon's start and stop (server.ts start, #stop): they own the sampler's timer, as described above.

Rules in play

• architecture.md rules 9, 11, 12, 14 and 15: a new port with a fake; the read bound; the sampler's states on every exit; the module's surface; nothing required imports it.
• safety.md "wire input is a claim": a gateway parses this block from every worker, so the schema bounds it.
• testing.md: every test below must fail if its claim breaks.

Tests

The seams are the adapters, the factory and the sampler directly, startDaemon for status and the lifecycle, formatStatus through the CLI, and src/lint-rules.test.ts for the lint. Model them on src/ports/host-info.test.ts, src/core/host-facts.test.ts, src/daemon/main.test.ts (the host facts status tests), src/cli/index.test.ts ("prints the host line") and src/lint-rules.test.ts ("fails a core file importing fs, child_process or a driver").

• the macOS probe reads total, available, swap and pressure from one bounded sysctl call, with available as hw.memsize × level ÷ 100 and swap from the used = field
• the macOS probe maps pressure 1, 2 and 4 to normal, warn and critical, and leaves only pressure out for any other line
• the macOS probe leaves memory out, and still reads CPU and disk, when sysctl exits non-zero, times out, or prints an unparsable size, level or swap line
• the macOS probe counts busy CPU time from cpus() as user, nice, sys and irq, with the core count
• the Linux probe counts busy CPU time from /proc/stat as everything but idle and iowait, against os.cpus().length cores
• the Linux probe reads memory from /proc/meminfo, with limited false and no pressure, and swap used as SwapTotal minus SwapFree
• the Linux probe leaves out the part whose file is missing or unparsable, and still reads the others
• both probes read free and total disk on the data folder through diskSpace, and leave disk out when it throws
• NodeFilesystem.diskSpace answers free and total bytes for a real folder, with total not below free
• the production factory builds the macOS probe on darwin and the Linux probe elsewhere, with a process runner of its own
• the sampler reads at start, then every 15 s, and skips a tick while a read is in flight
• the sampler gives up on a read after 10 s, ignores its late answer, and starts no other read until it settles
• the sampler computes busy % from two successive reads, and has no CPU after the first read
• the sampler leaves CPU out when Δtotal is 0 or less, or when the core count or limited changed between reads
• the sampler clamps busy % to 0–100
• the sampler never carries a part forward: one that fails is absent from that reading
• the sampler logs nothing for a part never read, one warn when a read part fails, and one when it is read again
• current() reports ageMs as the time since the read, 0 when the clock stepped back, and nothing before the first read finishes
• after dispose() the sampler holds no timer, and a read in flight changes nothing and logs nothing
• no file outside src/host-load/ imports a file inside it other than index.ts
• lint fails a host-load file importing node:fs/promises, child_process or core, and a core, leasing, driver, gateway or contract file importing host-load
• status.get load round-trips field for field, and is refused with a negative figure, an ageMs over one day, a busy % over 100, cores of 0 or an unknown pressure
• fitHostLoad caps ageMs at one day and drops a part whose figures break the schema, keeping the others
• a worker's status.get answers with load from the probe, on the starting answer and the running one
• a worker's status.get answers without load before the first read finishes, without waiting for it
• with hostLoadProbe: null a worker's status.get answers the same as with a probe, apart from load, and no host load timer is armed
• the daemon starts the sampler only after it holds its socket
• a daemon whose start fails after the sampler started (the admin secret cannot be written) leaves no host load timer pending
• the sampler treats a probe that rejects as a failed read, logs it once, and reads again on the next tick
• the sampler logs no CPU warn for a reading that has CPU counters but no busy %
• a stopped daemon leaves no host load timer pending
• simlock status prints the Load line after the Host line, on a starting daemon and a running one
• simlock status prints CPU unavailable, memory unavailable or disk unavailable for an absent part, and no swap or pressure part when memory is absent
• simlock status prints no pressure part when the reading has none, and limit on a limited CPU or memory
• simlock status formats bytes, busy %, cores and age as every row of the inputs table in this issue says
• e2e (e2e/host-load.test.ts, the ordinary lane, so Linux CI): a spawned worker's simlock status prints a Load: line with no pressure part, and its status --json load has memory without pressure, disk, and CPU after 16 s
• e2e/status-while-starting.test.ts accepts load beside daemon and host on a starting worker
• simlock status prints no Load line when the answer has no load block
• slow lane, on a Mac (e2e/slow-host-load.test.ts, tagged slow, skipped off darwin like e2e/slow-ios-smoke.test.ts): a running worker's status --json load has memory total equal to sysctl -n hw.memsize, a pressure, a disk total equal to the data folder's statfs, and after 16 s a busy % from 0 to 100

Done when

• On a Mac, the slow-lane test e2e/slow-host-load.test.ts passes through scripts/slow-e2e.sh.
• On Linux CI, e2e/host-load.test.ts passes: a real worker prints the Load: line, with no pressure.
• simlock status --json carries load with ageMs, cpu, memory and disk.
• Every test above is green, and the protocol is one above the base branch's.

Out of scope

• Linux cgroup limits (Linux load is measured against a cgroup v2 CPU or memory limit and marked limit #532). limited is always false on Linux here.
• The gateway: its own load, the worker view, simlock worker list (A gateway carries each worker's load in its view and reports its own #533).
• The console (The console shows each worker's load on its card and detail page #534).

Depends on

None. ADR 0028 (#529) must be accepted first, and the feature's feature:ready waits on it. By the maintainer's choice, this task runs alongside #519, #513, #348 and #318 with no ordering.

Approval

• Approved for delivery

Written by an agent.

Generated by Claude Code