Operational metrics kept over hours and shown in the console #623

Request: #612

Problem

The console can only show what happened in the last hour, and only what it saw itself. The daemon keeps no figures: the console builds its charts by counting the events it holds, so a reload starts them from zero. Several figures an operator needs are not sent at all: how long requests wait, why a waiting request is not placed, and why a device was quarantined.

Working the figures out from the event history on every read does not scale. The history can hold about a million events, and one full read takes more than a second. It also keeps only seven days.

Who it is for

Operators of a device cloud, one host or a gateway with workers. They want to know whether they have enough capacity, why agents wait, and why a device was taken out of use. A team lead deciding whether to buy another Mac. Someone tuning capacity, the warm pool or lease times who wants to see the effect.

Outcome

• The Overview shows how long requests waited (95th percentile) and how many leases were granted, for a chosen range: the last hour, 6 hours, 24 hours or 7 days. The same range drives three charts: leases held, requests in the queue, and device slots in use. A reload or a daemon restart shows the same figures.
• The Waiting page shows wait p95 for the last hour. Its inspector says, for a waiting request, why it is not placed yet. On a gateway there is one line per worker ("Full: running limit", "Drained", "OS not installed"); on a single host there is one line, the host. simlock list --requests prints the same reason.
• The Attention inspector says why a device was quarantined: what failed ("Purge failed", "Shutdown failed", "Delete failed", "Stalled while provisioning") and the tool's short error message. It still says so after a restart.
• simlock stats, GET /v1/stats and a new Usage page in the console show the full figures for a window. These are requests, grants and how each was served, wait, held time and turnaround, provisioning and boot time, rejections by reason, device incidents, and use per platform, per worker and per requester. A CSV export gives one row per request.
• Figures reach back 90 days by default. The event history itself is kept 7 days, as today.
• A 7-day figure comes back within half a second, even with 90 days of figures kept.
• Nothing slows a lease: recording the figures adds about 30 µs per event, written where the event is written today.

How it works for the user

flowchart LR
  O[Operator opens Overview] --> P{Range: 1h / 6h / 24h / 7d}
  P --> C[Tiles and 3 charts for that range]
  C -->|reload or daemon restart| C
  O --> W[Waiting inspector]
  W --> R["One reason per worker:<br/>Full: running limit / Drained / ..."]
  O --> A[Attention inspector]
  A --> Q["Purge failed:<br/>DriverError: simctl erase exited 164"]
  P -->|range older than the kept figures| N["Note: figures cover from &lt;time&gt;"]

Before:

$ simlock list --requests
Request req_8f2: tok_ci, iPhone 16, queued, waiting 42s

$ simlock stats
error: unknown command 'stats'

After:

$ simlock list --requests
Request req_8f2: tok_ci, iPhone 16, queued, waiting 42s — full: running limit for iOS

$ simlock stats --since 24h
Usage from 2026-10-15T13:00:00.000Z to 2026-10-16T13:00:00.000Z
  Requests:     214 (209 granted, 5 rejected)
  Sources:      warm 160, booted 41, provisioned 8
  Wait:         p50 400ms, p95 38s, max 2m10s (214 samples)
  Slots:        peak 8 of 8, mean 5.2
...

A history that was damaged at start (once):

$ simlock status
...
History: the event history was damaged and set aside at 2026-10-16 09:12; a fresh one was started.

Examples

The reason a waiting request gives, by what stops it:

Ranges asked for:

What could go wrong

• The figures could undercount after a crash or a full disk. Likely only on a full disk; the status says the history is not being written.
• A gateway's charts have gaps for the time a worker was disconnected. Expected; a gap shows as no data, not zero.
• A damaged history file is set aside and figures restart empty. Rare; the status says so once, and the old file is kept.
• After upgrading, figures start empty: the old event file is not imported.

Words used

• Event history: the record of every fact Simlock emitted, e.g. lease.granted. Kept 7 days.
• Figures: numbers worked out from the history, e.g. wait p95 for the last 24 hours. Kept 90 days.
• p95 (95th percentile): the wait that 95 of 100 requests stayed under, e.g. "p95 38s".
• Range / window: the span the figures cover, counted back from now, e.g. "last 24 hours".
• Reason (waiting): the first check that stopped a waiting request at its last placement attempt, e.g. "Full: running limit for iOS".
• Quarantine: a device taken out of use after cleaning it failed; Simlock retries and gives up after three tries.
• Set aside: a damaged history file renamed to history.db.damaged-<time> and kept, never deleted.

Non-goals

• Moving the registry, tokens or worker list into SQLite (Daemon state stores in SQLite (state.db) #624).
• Importing the old event file on upgrade.
• Alerting, thresholds, notifications, billing.
• A Prometheus or OpenMetrics endpoint.
• Explicit start and end dates in the console; the CLI and the route take them.
• Measuring what happens inside a device.
• Recording anything about a person beyond the requester id and token label events already carry.

Completion conditions

• On the fake driver, after a scripted run, the Overview shows wait p95 and leases granted, and three charts for the picked range (1h, 6h, 24h, 7d). After a reload and after simlock daemon stop and start, the same range shows the same numbers.
• The Waiting page shows wait p95 for the last hour. For a request waiting on a full single host, its inspector shows one line with the capacity reason. On a gateway with two workers, one drained, it shows two lines, one saying Drained. simlock list --requests prints the same reason.
• A device quarantined after a failed purge shows Purge failed and the error text, cut to 300 characters, in the Attention inspector, in simlock status --json, and after a daemon restart.
• simlock stats --since 24h --json, GET /v1/stats?since=24h and the Usage page for 24 hours show the same figures, and those match a count by hand from simlock events --since 24h.
• simlock stats --csv writes one row per lease request in the window.
• A fresh install writes history.db and no events.jsonl. simlock events --since 1h works with and without a running daemon.
• Events older than eventLog.retention are gone after the next prune; figures older than that are still answered until metrics.retention.
• A history.db corrupted while the daemon is stopped is renamed aside at the next start. Leasing works, and simlock status says the history was set aside.
• docs/CLI.md, docs/HTTP-API.md, docs/CONSOLE.md, docs/CONFIGURATION.md and docs/EVENTS.md describe the commands, the route, the pages, the keys and the history file.

Open questions

Decisions

• ADR 0032 — The event history is a SQLite database, with projections beside it (feat(history): history.db, usage projections, simlock stats, GET /v1/stats and console figures (#623) #626): history.db replaces events.jsonl; each event and its projection rows commit in one transaction; events 7 days, figures 90 days; a damaged file is set aside; a broken projection is rebuilt at start; a gateway keeps its own. Supersedes ADR 0016 §1 and §4, and narrows ADR 0006 §2.
• ADR 0016 §2, §3, §5, §7, §8 and ADR 0021 §5 still hold: the event facts the figures read, coversFrom and HISTORY_NOT_KEPT, labels, bucket widths, and how a gateway counts fleet requests.
• Not ADRs, decided in this session: the reason list below, the four quarantine kinds, and a worker's NO_CAPACITY answer carrying its capacity reason.
• Supersedes feature Usage metrics: utilisation, wait times and provisioning durations from the event history #329 and its open tasks usage.get and simlock stats: the figures from the event history, on a worker and a gateway #347, GET /v1/stats and the simlock stats --csv ledger #348 and Console Usage view with utilisation and waiting charts #349 (draft PR feat(usage): usage.get and simlock stats, the figures from the event history #356). Their figures are carried into this feature.

Shared values

Tasks

flowchart LR
  T7["#635 quarantine cause"] --> T8["#636 why waiting: worker"] --> T9["#637 why waiting: gateway"] --> T1["#629 history.db"]
  T1 --> T2["#630 projections"] --> T3["#631 usage.get + stats"] --> T4["#632 /v1/stats + CSV"]
  T4 --> T5["#633 Overview range + charts"] --> T6["#634 Usage page"]

Seven tasks change contract shapes and each bumps the protocol, so they run one after another (#635 → #636 → #637 → #629 → #630 → #631 → #632). The small ones go first, by the maintainer's choice. #633 and #634 change only the console.

• The event history is a SQLite file, history.db #629 The event history is a SQLite file, history.db — after it, the event history lives in history.db, read by index, set aside when damaged.
  Risk: leasing keeps working but events stop being recorded, and simlock events shows nothing new.
• Projections: request, minute and device fact rows #630 Projections: request, minute and device fact rows — after it, figures are kept 90 days in rows written with each event.
  Risk: figures are wrong or missing while leases work.
• usage.get and simlock stats from the projections #631 usage.get and simlock stats from the projections — after it, simlock stats prints the figures for any window.
  Risk: simlock stats prints wrong numbers or fails.
• GET /v1/stats and the simlock stats --csv ledger #632 GET /v1/stats and the simlock stats --csv ledger — after it, GET /v1/stats and a CSV export give the same figures.
  Risk: the route or the CSV disagrees with the CLI.
• Overview and Waiting: a range picker, wait p95, leases granted and three charts #633 Overview and Waiting: a range picker, wait p95, leases granted and three charts — after it, the Overview shows wait p95, leases granted and three charts for 1h to 7d.
  Risk: the Overview shows empty or stale charts.
• Console Usage page #634 Console Usage page — after it, the console's Usage page shows every figure for a range.
  Risk: the Usage page shows numbers that differ from simlock stats.
• A quarantined device says what failed #635 A quarantined device says what failed — after it, a quarantined device says what failed.
  Risk: a quarantined device shows no or the wrong cause.
• A waiting request says why it is not placed, on a worker #636 A waiting request says why it is not placed, on a worker — after it, a waiting request on a host says why it waits.
  Risk: a waiting request on a single host shows no or the wrong reason.
• A waiting request says why each worker cannot take it, on a gateway #637 A waiting request says why each worker cannot take it, on a gateway — after it, a waiting request on a gateway says why each worker cannot take it.
  Risk: a gateway's per-worker lines are missing or wrong.

Completion conditions and the task that delivers each:

• Overview tiles, three charts, range picker, same after reload and restart → Overview and Waiting: a range picker, wait p95, leases granted and three charts #633.
• Waiting p95 → Overview and Waiting: a range picker, wait p95, leases granted and three charts #633. Single-host reason → A waiting request says why it is not placed, on a worker #636. Gateway per-worker reasons → A waiting request says why each worker cannot take it, on a gateway #637. CLI reason → A waiting request says why it is not placed, on a worker #636, A waiting request says why each worker cannot take it, on a gateway #637.
• Quarantine cause in the Attention inspector, status --json and after restart → A quarantined device says what failed #635.
• simlock stats, GET /v1/stats and the Usage page agree → usage.get and simlock stats from the projections #631, GET /v1/stats and the simlock stats --csv ledger #632, Console Usage page #634.
• CSV → GET /v1/stats and the simlock stats --csv ledger #632.
• history.db replaces events.jsonl; simlock events with and without a daemon → The event history is a SQLite file, history.db #629.
• Event retention and figure retention → The event history is a SQLite file, history.db #629, Projections: request, minute and device fact rows #630.
• Damaged file set aside → The event history is a SQLite file, history.db #629.
• Docs → each task for the part it adds.

Open work in the same area:

• Usage metrics: utilisation, wait times and provisioning durations from the event history #329, usage.get and simlock stats: the figures from the event history, on a worker and a gateway #347, GET /v1/stats and the simlock stats --csv ledger #348, Console Usage view with utilisation and waiting charts #349, PR feat(usage): usage.get and simlock stats, the figures from the event history #356 — the same figures from the event file. Superseded by this feature and closed. PR feat(usage): usage.get and simlock stats, the figures from the event history #356's figure code is reused where it fits (usage.get and simlock stats from the projections #631).
• Daemon state stores in SQLite (state.db) #624 — state stores in SQLite. Will use the SQLite port The event history is a SQLite file, history.db #629 adds. No Depends on: it is a request, not yet specified.
• Gateway republishes a worker-pushed request.dispatched into its history #434 — a gateway republishes a worker-pushed request.dispatched into its history. Projections: request, minute and device fact rows #630 never reads request.dispatched as a request fact; fleet outcomes come from the gateway's own request.granted and lease.rejected only (ADR 0021 §5). No Depends on.
• Console fleet totals count a starting worker's leases and devices as zero #406, A gateway hides an owner's leases on a worker that is starting #405 — gateway fleet totals during a worker start. They touch the Overview's current-state tiles, not the range figures Overview and Waiting: a range picker, wait p95, leases granted and three charts #633 adds. No Depends on.
• Every device shape says when the device entered its state #318 / PR feat(core): every device shape says when the device entered its state #324 — device shape gains when it entered its state. Same schema as A quarantined device says what failed #635, different fields. They run at once; whichever merges second takes the base branch.
• Console: time in state for every device #319 — console time in state. Changes the Attention inspector's device facts, as A quarantined device says what failed #635 does. It is a draft; no Depends on.
• A request waits only behind requests for the same platform #387 — a request waits only behind same-platform requests. It would change when A waiting request says why it is not placed, on a worker #636 gives behind-earlier-request. A request, not specified; no Depends on.
• A waiter that times out while its lease is being written gets a timeout, and the lease is kept #402 — a waiter timing out mid-grant. Gives one wrong wait sample at that edge; no Depends on.
• Android: stream a leased physical phone's screen #618, iOS: stream a leased physical iPhone's screen #619 — physical screen streams. Touch the Leases inspector and stream module, not these screens. No Depends on.

Written by an agent.