Live screen streams of leased devices #593

Request: none

Delivery: one PR, #576 (branch feature/stream-poc), one review over the whole diff, no task split: the maintainer's choice, 2026-10-15. File and line references below are at f68c1c5f unless marked (main).

Where it stands: unit and console-lane tests pass; the single-worker console check on real devices passed on 2026-10-15 (iPhone 17 simulator, iOS 27; Pixel 9 emulator); an in-process gateway and worker stream through each other in the fast lane (F30); the slow-lane tests and the gateway check on real devices have not passed yet. No slow-lane test covers a gateway.

Problem

An operator cannot see what is on a leased device. An agent drives a simulator or an emulator on a worker, maybe a machine in another room, behind a gateway. When the agent hangs, loops or fails a test, the only way to see the screen is to log in to that machine and find the right window. Nothing in Simlock shows a picture.

Who it is for

• An operator watching agents' devices in the console, on any lease in the fleet.
• An agent or a tool that holds a lease and wants a live picture of its own device over the HTTP API, for example to record a failing run.

Outcome

• In the console, a lease's page has a Screen panel. Pressing Show screen shows the device's screen live, with its size, rate and bandwidth over the last two seconds. Pressing Stop, leaving the page or closing the tab stops it.
• Over HTTP, one GET on a lease's stream URL returns the live picture for as long as the connection stays open.
• This works for leases on iOS simulators and on Android emulators, on a single worker and through a gateway, with the same URL and the same bytes.
• Nothing is captured while nobody looks. A capture never outlives its viewer or its lease, and a broken capture ends instead of freezing, with two accepted exceptions: after Simlock reboots a crashed iOS simulator under its lease, the viewer can see the old picture until Stop and Show screen (F31); and an Android open still pushing (≤ 30 s) or starting (≤ 15 s) when lease.endWaitMs runs out can overlap the reclaim. A still screen stays live.

How it works for the user

sequenceDiagram
  participant U as Operator (console)
  participant G as Gateway
  participant W as Worker
  participant D as Device
  U->>G: Show screen (GET /v1/stream/leases/L1)
  G->>G: lease L1 known, caller may see it?
  alt unknown, or another agent's lease
    G-->>U: 404 UNKNOWN_LEASE
  else worker disconnected
    G-->>U: 503 WORKER_UNREACHABLE
  end
  G->>W: relay to the worker that holds L1
  W->>W: lease still held by that requester?
  W->>D: start capture
  alt physical device
    W-->>U: 415 STREAM_NOT_SUPPORTED
  else capture program missing, or no picture in 30 s
    W-->>U: 503 STREAM_UNAVAILABLE
  end
  W-->>U: 200, picture headers, then frames
  U->>G: Stop / leave page (request closed)
  G->>W: relay cancelled
  W->>D: capture stopped
  Note over W,D: the lease ends while watched: capture stopped, the stream ends

Before (any daemon today):

$ curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $TOKEN" \
    http://gw:7777/v1/stream/leases/L1
404

After (stream.enabled: true on the gateway and the worker):

$ curl -sD - -o screen.bin -H "Authorization: Bearer $TOKEN" \
    http://gw:7777/v1/stream/leases/L1
HTTP/1.1 200 OK
cache-control: no-store
content-type: application/vnd.simlock.stream+h264
simlock-stream-codec: h264
simlock-stream-fps: 30
simlock-stream-height: 720
simlock-stream-width: 330
x-simlock-lease: L1
^C        # closing the request stops the capture on the worker

$ curl -s -H "Authorization: Bearer $TOKEN" http://gw:7777/v1/stream/leases/L9
{"error":{"code":"STREAM_NOT_SUPPORTED","message":"A physical iPhone's screen cannot be streamed; only a simulator's can"}}

Examples

What could go wrong

• A picture freezes and looks live. Unlikely: a stream that goes 30 s without a byte ends, and a still Android screen sends a small "alive" frame every 5 s. One known case: after Simlock reboots a crashed iOS simulator under its lease, the viewer can see the old picture until Stop and Show screen.
• A capture keeps running after its lease ends, on a device someone else now holds. Unlikely: the lease end stops it before the device is reclaimed.
• An agent sees another agent's screen. Unlikely: ownership is checked on the gateway and again on the worker.
• An Android stream is slow or choppy (about 21 fps while scrolling, measured under heavy host load). Likely on busy machines.
• A worker's CPU rises while a screen is watched: one capture per viewer.

Words used

• stream: one open GET /v1/stream/leases/{leaseId} and the picture it carries, e.g. the console's Screen panel while it is on.
• capture: what films one device for one stream, e.g. one simlock-screen-capture process per viewer of a simulator, or one scrcpy server inside an emulator.
• envelope: the body's framing: 4-byte big-endian length (counting the tag), 1-byte tag, payload. Tags: 0x05 JSON meta (always first), 0x01 codec description (avcC), 0x02 key frame, 0x03 delta frame, 0x04 JPEG still, 0x06 alive (empty). A reader skips a tag it does not know.
• alive frame: an empty frame a capture sends when the screen has not changed, so a still screen is not mistaken for a dead capture.
• key frame: a frame a decoder can start from; a delta frame needs the frames before it.
• relay target: a name a gateway sends a relayed HTTP exchange to on a worker, e.g. agent-device, and now stream.
• capability: an optional module a worker reports with a state, e.g. stream: available (no reason) or stream: unavailable (ios: No screen capture program at …). A worker with streaming off reports no stream row.
• scrcpy server: the small Android program from the scrcpy project that encodes an Android screen on the device; Simlock carries it unchanged.
• smart socket: adb's own way to reach a socket inside a device through the adb server (host:transport:<serial>, then localabstract:<name>), with no host port opened.

Non-goals

• Input. No touches, keys or text through the stream; v1 is view-only.
• Physical devices, iOS and Android. Refused in v1.
• Audio.
• Recording or saving streams on the worker, and any history of who watched.
• A CLI command. The console and the HTTP API only.
• Choosing a device by whether it can be streamed (requires), and showing "being watched" anywhere.
• Publishing Simlock or the iOS capture program as a prebuilt package.
• A GPU against software encoder measurement for Android emulators.

Completion conditions

• With stream.enabled: true, an operator presses Show screen on a leased iOS simulator in the console, on a single worker, and sees the screen move as an agent drives it. Manual check by the maintainer; the PR's Done when gives the date and devices (done 2026-10-15, iPhone 17 / iOS 27).
• The same on a leased Android emulator. Manual check by the maintainer (done 2026-10-15, Pixel 9).
• The same through a gateway, for both platforms. Manual check by the maintainer on real devices, shown with date and devices in the PR's Done when (not done yet; no slow-lane test covers a gateway). The maintainer accepted this as a manual check on 2026-10-15.
• GET /v1/stream/leases/{leaseId} answers each row of Examples with the status and code shown, on a worker and through a gateway.
• Closing the request ends the capture on the worker: no capture process, and on Android no scrcpy app_process and no simlock-scrcpy-*.jar in /data/local/tmp. Real devices: e2e/slow-ios-stream.test.ts, e2e/slow-android-stream.test.ts.
• Releasing a lease while it is watched ends the stream, and leaves the same nothing behind. Real devices: the same two tests.
• A still screen keeps its stream open past the 30 s silence bound. Real devices: e2e/slow-android-stream.test.ts (45 s, an alive frame required), e2e/slow-ios-stream.test.ts (35 s, frames required).
• With stream.enabled: false, no stream route exists and the worker reports no stream capability.
• pnpm build copies the scrcpy server after checking its pinned SHA-256 (a mismatch fails the build), and on macOS with Xcode builds the iOS capture program; nothing is downloaded to stream.
• The capture tool and the scrcpy server sit under vendor/ with their license texts and a NOTICE.md each, and the check that enforces it passes.
• The slow-lane tests e2e/slow-ios-stream.test.ts and e2e/slow-android-stream.test.ts pass.
• A lease that ends while a capture is still opening stops that open, and its reclaim waits for the open to settle and what it started to close, up to lease.endWaitMs (default 600000 ms). Past that the wipe starts anyway, and an Android open still pushing (≤ 30 s) or starting its server (≤ 15 s) can overlap the reclaim; accepted, with that bound.

Open questions

None.

Decisions

• ADR 0031 — A screen stream is an optional module, and a driver films the device: the route, the envelope, close-is-stop, stoppable opens, the driver capability, both platforms, the relay and its idle bound, no limit on captures per lease in v1, no business events, vendored code. Accepted — not yet implemented, PR docs(adr): 0031 a screen stream is an optional module, and a driver films the device #594.
• One PR and one review (feat(stream): a leased simulator's or emulator's screen, streamed to whoever is looking at it #576), no task split: the maintainer's choice, 2026-10-15.
• Known gaps accepted for v1: an iOS picture can freeze after a reboot under its lease until Stop and Show screen (F31); an Android open still running when lease.endWaitMs runs out can overlap the reclaim by up to 30 s pushing or 15 s starting.
• The gateway on real devices stays a manual maintainer check (accepted 2026-10-15).
• ADR 0024 — Leasing calls lease-end participants before reclaim: how a lease end stops captures.
• ADR 0026, ADR 0028 — the relay over the uplink and its byte channels; ADR 0031 amends 0028 §4: the 330 s deadline covers the channel and the response head, then a 330 s idle timer bounds every relayed body.
• ADR 0030 (docs(adr): 0030 optional modules share one contract and load only when turned on #592, not merged) is not a dependency: daemon/main.ts wires the module's parts as it does agent-device's; if 0030 is accepted the module moves onto its contract.

Technical spec

Facts

Modules touched

flowchart LR
  MAIN["daemon/main.ts *"] --> ST["stream/ (new) *"]
  ST --> LP["leasing: LeaseEndParticipant"]
  ST --> CAT["core: DriverCatalog, screen.ts, config.ts *"]
  IOS["drivers/ios *"] --> CAT
  AND["drivers/android/screen *"] --> CAT
  IOS --> P["ports: BytePipeLauncher, TcpStreamConnector *"]
  AND --> P
  ST -. gateway relay .-> GW["gateway/relay.ts *"]
  UI["ui/src/screen, views/leases.tsx *"] -. HTTP .-> ST

(* = new or changed.) The gateway sequence is ADR 0031 §7's diagram. Also changed: src/contract (schema, protocol), src/http/errors.ts (405, 415, 502), src/drivers/android/adb-registrar.ts and adb-host-protocol.ts (shared framing), src/drivers/android/index.ts (wires screenCapture), src/drivers/android/screen/AGENTS.md (new manual), .dependency-cruiser.mjs (not-stream, stream-not-edges), package.json (build:capture), scripts/build-capture.mjs, vendor/, e2e/fake-driver.

Surface and extension points

• src/stream/index.ts: createWorkerStream → { mount, relayTarget, relayTargetName, leaseEnd, capabilitySource }; createGatewayStream → { mount, relayTargetName }.
• src/core/screen.ts: ScreenCapture, ScreenFeed, ScreenNotSupportedError (415), ScreenUnavailableError (503), SCREEN_FRAME_TAG, encodeScreenFrame, encodeScreenMeta. Two implementations exist (iOS, Android); the registry is DriverCatalog.
• src/ports: BytePipeLauncher (binary stdout, stdin as liveness) and TcpStreamConnector, each with a Node adapter and a fake.

Contract and event changes

• No new contract operation. configSchema gains an optional stream object: protocol +1 (36 → 37, F24).
• New HTTP route GET /v1/stream/leases/{leaseId}; statuses 405, 415, 502 join HttpStatus; codes STREAM_NOT_SUPPORTED (415) and STREAM_UNAVAILABLE (503).
• Events: none. Both EVENTS.md files say a stream emits no event and why (F27).

Other code on the same state

• Leasing's lease end and reclaim (F1, F6): the participant stops opens in progress, waits for them to settle, closes every capture of that lease and waits, bounded by the driver's bounds and leasing's signal. Lease end wins. A nuke or daemon shutdown cuts the wait short, so its wipe may run while a capture closes: accepted, maintenance is an operator's reset.
• The health monitor rebooting a leased device: on Android the stream ends when the device socket closes or at the 30 s silence bound, and close cleans up against the serial stored at open. On iOS the viewer can see the old picture until Stop and Show screen (F31); the lease end stops it as usual.
• The Android driver's private adb server: the capture's push, shell and smart socket go through it, scoped to the emulator's serial. Daemon stop ends every viewer's request first, then disposes it (F26).
• The iOS driver's shutdown and erase: the lease end closes the capture before reclaim (F6); a simulator shut down under a live capture is the reboot case above (F31).
• A daemon stopped or killed: the iOS program sees its stdin close and stops; on Android the host adb shell ends with the daemon, which ends the device server (F21, F26).
• agent-device's relay target: a second name in the same table. The relay's new idle bound applies to it too (F12).
• The link's channels: a stream holds one channel for its life; at 64 open channels a new relay waits.

Rules in play

• architecture.md 1–3, 9, 11, 12, 15, 18–21: the core names no platform; ports for spawn, timers and sockets (the relay's idle timer on the Clock port, F12); every wait bounded; every capture ends in one state.
• safety.md 4, 9, 10: nothing downloaded; Simlock's own adb server for emulators; worker answers and headers are claims.
• events.md: no events, and the catalogue says so.
• vendor-and-third-party-code.md: lands with this feature.
• documentation.md: docs/HTTP-API.md, docs/CONFIGURATION.md, docs/CONSOLE.md, and the src/stream/AGENTS.md manual.

Tests

Seam: the stream links over a test registry and scripted feeds (src/stream/worker-link.test.ts), the gateway link over a fake fleet (src/stream/gateway-link.test.ts), drivers' screen code over fake ports (src/drivers/android/screen/session.test.ts, src/drivers/ios/screen-capture.test.ts), the console lane (e2e/console/screen.spec.ts), the slow lane on real devices.

• each row of Examples answers the status and code shown, on a worker and through a gateway.
• a gateway answers a disconnected worker 503 WORKER_UNREACHABLE, and a worker with no stream relay target 503 STREAM_UNAVAILABLE naming the worker.
• a lease that ends while its capture is starting tells the driver to stop opening, waits for the open to settle, closes what attached, and answers UNKNOWN_LEASE (worker-link.test.ts, captures.test.ts).
• a request aborted while the driver is still opening closes the capture as soon as it attaches.
• a lease that ends after the viewer left, while the open is still running, still waits for that open (worker-link.test.ts).
• through a real in-process gateway and worker: a relayed stream answers 200 with headers and meta; releasing the lease on the gateway stops the capture on the worker; "a viewer leaving the gateway stops the capture on the worker"; a worker that does not stream answers 503 STREAM_UNAVAILABLE naming it (src/daemon/stream-gateway.e2e.test.ts).
• two viewers of one lease get two captures; one leaving keeps the other; the lease ending stops both (worker-link.test.ts).
• a lease that ends while watched ends the stream and closes the driver's screen; the participant returns only after the close, or at leasing's signal.
• a capture with no meta frame in 30 s, or silent 30 s after one, ends with an error and is closed, on a fake clock.
• the response carries the picture in its headers (src/stream/service.test.ts).
• a relayed body that keeps sending outlives the 330 s deadline; one silent for the idle time ends WORKER_UNREACHABLE with relay.cancel; a head that does not arrive fails at the deadline.
• a worker refusal body is read to 16 KiB only; a code that does not match the shape reaches the caller as STREAM_UNAVAILABLE; a 2xx without a picture is 502; a relay that cannot reach the worker is 503 with the relay's code, and a worker with no stream is 503 STREAM_UNAVAILABLE naming it (gateway-link.test.ts).
• a relayed picture whose codec or size fails checkedMeta is refused STREAM_UNAVAILABLE.
• streaming on, simlock worker list --json shows stream available ("shows nothing and starts no capture until asked"); off, it shows no stream capability and the route answers 404 ("a worker with streaming off refuses the picture and says what to do"), both in e2e/console/screen.spec.ts.
• the capability is available with no reason when any platform can capture, else unavailable with each platform's reason (create.test.ts).
• pnpm build fails, copying nothing, on a scrcpy jar that misses its pin, and copies one that matches byte for byte (src/build-capture.test.ts).
• iOS: the program runs on the UDID inside Simlock's device set with the settings; physical is not supported; a missing program is unavailable; a stopped open starts nothing; close stops it through stdin, then SIGKILL after 2 s.
• Android: per-session scid, jar path and socket; the device socket reached by host:transport and localabstract with no adb forward; the socket read only when pulled, every delta delivered; alive on a still screen, and none before the first picture; an open stopped during the push starts no server and removes the jar; physical not supported; close ≤ 5 s kills the shell and removes the jar.
• the console's tag table equals core's, and its rate covers the last two seconds.
• console lane (e2e/console/screen.spec.ts): "shows nothing and starts no capture until asked"; "Show screen starts the capture, paints it, and reports what it is showing"; "Stop stops the capture, and so does leaving the page"; "a worker with streaming off refuses the picture and says what to do"; "a release takes the pane away and leaves no capture running".
• slow lane: e2e/slow-ios-stream.test.ts streams meta, an avcC description and a key frame, asserts the stream of a still simulator is still open at 35 s, and a viewer leaving and a lease released under a viewer each stop the capture; e2e/slow-android-stream.test.ts does the same, holds a still screen 45 s with alive frames, checks no server and no jar on the device after the viewer leaves the still screen, then releases a lease under a fresh viewer.

Written by an agent.