GET /v1/stats and the simlock stats --csv ledger #632

Part of #623.

Scope

After this PR, GET /v1/stats answers the same figures as simlock stats --json for the same window, with an operator token, on a single host and on a gateway. simlock stats --csv writes one row per lease request in the window, a file a spreadsheet opens. Both read the ledger from the request rows #630 writes, through usage.get (#631).

In short

A dashboard or script fetches GET /v1/stats?since=24h and gets the figures as JSON. &ledger=1 adds requests[], one entry per request. An operator runs simlock stats --since 7d --csv > week.csv and opens it in a spreadsheet.

A ledger entry is one request's life: when it was asked, granted or refused, and ended, and what it waited and held, e.g. req_8f2, tok_ci, ci-runner, ios, iPhone 16, …, granted, false. The ledger holds at most 100,000 entries, the newest; a longer window is cut and says so.

flowchart LR
  H["GET /v1/stats (new)"] --> W[[usage window rules<br/>contract, #631]]
  CLI["simlock stats --csv (new flag)"] --> W
  W --> U["usage.get + includeLedger (changed)"]
  U --> R[(request rows, #630)]
  CLI --> CSV[["CSV writer (new)"]]

Session, before and after:

$ curl -s -H "Authorization: Bearer $OP" 'http://127.0.0.1:4700/v1/stats?since=1h'
404                                                  # today: no route

$ curl -s -H "Authorization: Bearer $OP" 'http://127.0.0.1:4700/v1/stats?since=1h'
{"window":{"from":…,"to":…},"coversFrom":…,"partial":false,"bucketMs":60000,"totals":{…},…}

$ simlock stats --since 1h --csv
requestId,requester,label,platform,model,workerId,source,requestedAt,grantedAt,rejectedAt,endedAt,waitMs,heldMs,turnaroundMs,outcome,probe,reason,endReason
req_8f2,tok_ci,ci-runner,ios,iPhone 16,,warm,2026-10-16T12:01:02.000Z,2026-10-16T12:01:02.400Z,,2026-10-16T12:09:30.000Z,400,507600,508000,granted,false,,explicit

$ simlock stats --since 90d --csv > q.csv      # a window with more than 100,000 requests
{"notice":"The ledger holds the newest 100000 requests of the window; ask for a shorter window for the rest."}

Inputs:

The flag names in a message follow the surface: --since on the CLI, since on the route; the rule behind both is one function.

If this goes wrong, the route and the CLI give different numbers for one window, or the CSV has a row count that does not match simlock events.

Technical spec

Facts

Modules touched

Searches: /v1/stats, includeLedger, text/csv (0 hits on main); usage.get (#631's files); badRequest in src/http/app.ts.

• src/contract/schemas.ts, operations.ts — usage.get input gains includeLedger?: boolean; output gains requests?: LedgerEntry[]. protocol.ts +1.
• src/contract/usage-window.ts (usage.get and simlock stats from the projections #631) — called with the query; the route words its refusal (F14).
• src/core/usage/ — ledger entries from request rows, using usage.get and simlock stats from the projections #631's wait, held and turnaround functions.
• src/http/app.ts — the route.
• src/daemon/dispatcher.ts, src/gateway/dispatcher.ts — pass includeLedger to the reader; on a gateway the ledger's labels come through the same prefix strip (F17).
• src/cli/index.ts (--csv), src/cli/csv.ts (new, the RFC 4180 writer).
• Docs: docs/HTTP-API.md (route under "Operator routes", 400, 422 and 503 rows of Errors), docs/CLI.md (--csv), docs/CLIENT.md (includeLedger). Manuals: contract (protocol), http.
• Tests: src/contract/operations.test.ts, src/core/usage/*.test.ts, src/core/usage/usage-read.timing.test.ts (usage.get and simlock stats from the projections #631), src/gateway/dispatcher.test.ts, src/http/app.test.ts, src/cli/csv.test.ts, src/cli/index.test.ts, e2e/http-api.test.ts, e2e/usage-stats.test.ts.

flowchart LR
  http[http/app*] --> win[contract/usage-window*]
  cli[cli*] --> win
  cli --> csv[cli/csv* new]
  http --> disp[daemon or gateway dispatcher]
  cli --> client[simlock-client] --> disp
  disp --> ur[core/usage*]
  ur --> rows[#630 request rows]

sequenceDiagram
  participant C as HTTP client
  participant A as http/app
  participant D as dispatcher (in process)
  C->>A: GET /v1/stats?since=1h&ledger=1
  A->>A: window rules (400 on a bad parameter)
  A->>D: usage.get { from, to, includeLedger: true }
  D->>D: role (403 for agent), parse (400)
  alt window ends before oldest row
    D-->>A: HISTORY_NOT_KEPT { oldestTs }
    A-->>C: 422 { error: { code, message, oldestTs } }
  else
    D-->>A: output with requests[]
    A-->>C: 200 output
  end

Contract and event changes

This PR bumps the protocol by one.

• usage.get input gains includeLedger: boolean, optional, default false. A non-boolean is BAD_REQUEST (feature Operational metrics kept over hours and shown in the console #623 Shared values, "usage.get").
• usage.get output gains requests[] and ledgerTruncated?: true, present only when includeLedger is true. One entry per request row in the window, probes and refused-at-admission rows included, by requestedAt, then requestId:
  { requestId, requester, label?, platform, model?, workerId?, source?, requestedAt, grantedAt?, rejectedAt?, endedAt?, waitMs?, heldMs?, turnaroundMs?, outcome: "granted" | "rejected" | "declined" | "open", probe: boolean, reason?, endReason? } (F11, with rejectedAt, probe and declined added).
  • outcome: granted with grantedAt by to; declined for a probe with rejectedAt by to, the worker's lease.declined (F15); rejected for any other row with rejectedAt by to; else open.
  • At most 100,000 entries: when the window has more, the newest 100,000 are kept and ledgerTruncated is true; otherwise it is absent.
  • waitMs, heldMs and turnaroundMs are usage.get and simlock stats from the projections #631's samples: absent where usage.get and simlock stats from the projections #631 gives none (a probe's and a refused-at-admission row's wait).
  • The same window rule as usage.get and simlock stats from the projections #631's figures: a grant, rejection or end after to is unseen.
    • A grant after to: no grantedAt, source or workerId; the entry is open.
    • A rejection after to: no rejectedAt or reason; open.
    • An end after to: no endedAt, endReason, heldMs or turnaroundMs; still granted.
  • label from the token store, as usage.get and simlock stats from the projections #631's requesters.
• New route GET /v1/stats (feature Operational metrics kept over hours and shown in the console #623 Shared values, "GET /v1/stats"):
  • since: a duration the contract's parser takes (F1), else 400 Invalid duration: <value>.
  • from, to: strict ISO 8601 with Z or an offset (F14), else 400 Invalid time: <value> (use ISO 8601 with Z or an offset). A + must be sent as %2B; an unencoded one arrives as a space and is refused.
  • The other refusals, worded for the route: stats takes since or from, not both, to needs from, from must be earlier than to, or than now when to is left out.
  • ledger: absent or 1; any other value 400 ledger must be 1.
  • from without to runs to now.
  • The body is usage.get's output, unwrapped. Errors take their status from the error table (F4, F5).
• The dispatcher parses the output, ledger included, on the main thread; usage.get and simlock stats from the projections #631's timing test bounds a 100,000-entry answer at 2.5 s (Tests; 1 s in the first spec, measured 1.44 s on the CI runner and about 0.6 s locally — decided in delivery, pending the maintainer).
• simlock stats --csv: asks with includeLedger: true, writes the ledger to stdout as RFC 4180 CSV: one header row of the field names above in that order, one row per entry, CRLF line ends, timestamps as ISO 8601 UTC, durations as whole milliseconds (Math.round, no decimal point), probe as true/false, an empty cell for an absent value. A cell starting with =, +, - or @ gets a leading ', since requester ids and labels are free text (F8, F9). Then a cell holding a comma, a quote, a CR or an LF is quoted, its quotes doubled. When ledgerTruncated, the CLI writes the notice The ledger holds the newest 100000 requests of the window; ask for a shorter window for the rest. on stderr (F13) and exits 0.
• No event changes.

Other code on the same state

Searches: request rows (#630, read only here); usage.get (#631's handlers, reader). One reader: the route and the CLI call the same usage.get; nothing here writes. #631's read (#631 "Other code on the same state") covers #630's writes and prunes landing beside a ledger read: the ledger comes from the same read transaction on the read thread as the figures.

Failure modes

Nothing is written: a restart mid-call leaves nothing. Two callers for one window share #631's one read in flight.

Rules in play

• Architecture rule 10: "Never implement the same decision twice": the window rules are usage.get and simlock stats from the projections #631's usage-window.ts, called by the CLI and the route; wait, held and turnaround are usage.get and simlock stats from the projections #631's functions, used by figures and ledger.
• Architecture rule 8: the CLI formats the CSV; the daemon builds the entries.
• Safety rule 10: "Input that does not match something known is refused": since, from, to and ledger are parsed and bounded before dispatch; a time without a zone, or in any form but ISO 8601, is refused; an unknown ledger value is refused.
• Documentation rules 2 and 3: docs/HTTP-API.md names no ADR; error bodies name no repo path.
• Testing rule 2: each 400 test starts from an accepted query, then changes one parameter.

Tests

Seam: e2e/http-api.test.ts for the route over a real daemon; the route's query branches in src/http/app.test.ts, extending "GET /v1/events replays via events.replay, translating ?since to an absolute sinceTs"; src/cli/index.test.ts "CLI: stats" (#631). Ledger entries at #631's seam, the usage.get handler over a real history.db.

• usage.get with includeLedger lists one entry per request in the window, by requestedAt then requestId, with the timestamps and waitMs, heldMs and turnaroundMs of a granted and released request
• usage.get's ledger marks a request still waiting at to as open, one granted after to as open with no grantedAt, source or workerId, a lease open at to as granted with no endedAt, endReason, heldMs or turnaroundMs, a rejection as rejected with its rejectedAt and reason, and a declined probe on a worker as declined
• usage.get's ledger agrees with its figures: entries granted equal totals.granted, and a request's waitMs is a sample in totals.wait
• usage.get leaves requests out when includeLedger is absent or false, and refuses a non-boolean includeLedger as BAD_REQUEST
• usage.get's ledger carries the token label of a known requester and none for an unknown one, and lists a probe (probe: true) and a refused-at-admission rejection as entries
• usage.get's ledger over 100,001 requests holds the newest 100,000 with ledgerTruncated: true, and over exactly 100,000 holds them all with no ledgerTruncated
• simlock stats --csv writes the truncation notice on stderr and exits 0 when ledgerTruncated, and writes no notice otherwise
• simlock stats --csv on an error (HISTORY_NOT_KEPT) writes the error line on stderr, nothing on stdout, and exits 12
• the CSV writer quotes a field holding a comma, a quote, a CR or an LF, doubles its quotes, ends rows with CRLF, writes an empty cell for an absent value, and puts ' before a cell starting with =, +, - or @ (each of the four) and not before one starting otherwise
• simlock stats --csv prints the header row and one row per ledger entry with times as ISO 8601, durations as whole milliseconds (400, not 400.0), probe as true or false, and asks usage.get with includeLedger: true
• simlock stats refuses --csv with --json as a usage error, exit 2
• GET /v1/stats dispatches usage.get for since back from now, from to to, from to now, and the last 24 hours with no parameter
• GET /v1/stats answers 400 with the Inputs table's message for since with from, to without from, from not before to, a bad duration, the times 1, 2026, 10/15/2026, no zone and an unencoded +, and ledger other than 1, each after the same query without the fault answered 200, and answers 200 for %2B02:00
• GET /v1/stats passes includeLedger: true only for ledger=1
• GET /v1/stats answers HISTORY_NOT_KEPT with status 422 and oldestTs in the body, and HISTORY_UNAVAILABLE with 503
• the gateway handler with includeLedger lists the fleet's request rows as entries, labelled after stripping its own gw: prefix, and the worker handler passes includeLedger the same way
• (timing) a 100,000-entry ledger answer over usage.get and simlock stats from the projections #631's seeded rows returns through the dispatcher, output parse included, within 2.5 s
• (e2e) GET /v1/stats with an operator token returns the same body as simlock stats --json for a fixed from/to, without requests, and &ledger=1 adds them, on a single host and on a gateway with one worker; an agent token gets 403
• (e2e) simlock stats --csv --from <before the run> --to <after it> after a scripted run with an already-leased refusal has one data row per lease.requested in that window of simlock events, plus one for the refusal

Done when

• After a scripted run against the fake driver, GET /v1/stats?from=<before the run>&to=<after it> with an operator token returns the same figures as simlock stats --from <same> --to <same> --json, on a single host and on a gateway.
• simlock stats --csv --from <before the run> --to <after it> > run.csv writes one header row and one row per request row; a spreadsheet opens it with the columns apart.
• An agent token gets 403; a window older than the kept figures gets 422 with oldestTs in the body.
• docs/HTTP-API.md describes the route under "Operator routes" and its 400, 422 and 503 errors; docs/CLI.md describes --csv; docs/CLIENT.md describes includeLedger.

Out of scope

• The console's Usage page and its CSV button (Console Usage page #634).
• A Prometheus or OpenMetrics endpoint.

Depends on

• usage.get and simlock stats from the projections #631

Written by an agent.